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Executive Summary 

The 2004/ 09 Beginning Postsecondary Students Longitudinal Study (BPS:04/ 09), conducted 
for the U.S. Department of Education’s National Center for Education Statistics, collected 
information about students’ education and employment in the 6 years since they first enrolled in 
postsecondary education. 

This report describes the methodology and findings of the BPS:04/ 09 data collection, which 
included a student interview, a transcript data collection, and an administrative records match. 

Sample Design 

The target population 1 for the BPS:04/09 study was first-time beginners (FTBs): students 
who started their postsecondary education for the first time during the 2003-04 academic year at any 
postsecondary institution in the United States or Puerto Rico. All sampled students for the BPS:04 
cohort were FTBs who were eligible for the 2003-04 National Postsecondary Student Aid Study 
(NPSAS:04) and whose FTB status was confirmed in BPS:04/ 06, the first follow-up interview. 

The BPS:04/09 sample comprised all 18,640 students determined to be eligible during 
BPS:04/06. At the conclusion of the BPS:04/09 data collection, 16,680 had enough data from the 
student interview or from administrative sources to be classified as BPS:04/ 09 study respondents. A 
BPS:04/ 09 study respondent was defined as any sample member who was determined to be eligible 
for the study, was still alive at the time of the BPS:04/ 09 data collection, and had the requisite valid 
data from any source to allow construction of his or her enrollment history. 

Data Collection 

The data collection design for BPS:04/ 09 involved several stages. The initial process of 
locating sample members involved batch-locating activities to update sample members’ address and 
telephone information from several sources. In addition, sample members and their parents were 
sent an initial mailing to collect updated contact information. 

Once the initial round of locating was completed, sample members were sent information 
regarding study participation and informed consent. Data collection then began and was conducted 
in three phases. Sample members who completed interviews during the early response phase, or first 
phase, and the nonresponse conversion phase, or last phase, were offered an incentive of $30. 

Sample members who completed interviews during the production, or middle phase, were offered 
an incentive of $20. First follow-up (BPS:04/ 06) student interview nonrespondents were offered an 
additional $20 during each phase of data collection. 

Of the 18,610 sample members in the BPS:04/09 student interview data collection (cases 
found deceased before data collection were excluded), 16,920 (91 percent) were successfully located, 
and 15,160 either partially or fully completed an interview. The response rate was 82 percent among 
the eligible sample and was 90 percent among those sample members who were successfully located. 
The majority of completed interviews (9,630) were obtained in web mode, wherein respondents 
accessed and completed the interview online. 


1 The target population consists of all institutions or students in the survey population, and is the population to which inferences are 
made. 
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Executive Summary 


The BPS:04/ 09 student interview was designed as a web-based instrument to be used for 
web, telephone, and field respondents. This design required that several important features be 
embedded in the instmment, such as extensive help text to assist respondents and warnings to alert 
them when a response fell outside a predetermined range of likely responses. 

The BPS:04/ 09 interview took approximately 20 minutes to complete. On average, web 
respondents completed the interview in 19.4 minutes, telephone respondents completed the 
interview in 20.2 minutes, and field respondents completed the interview in 19.6 minutes. 

An evaluation of the quality of the data provided by the BPS:04/ 09 student interview 
showed that methodological features, such as help text and conversion text built into the instmment 
and training and supervision of interviewing staff, aided in the successful administration of the 
interview. 

Data collection quality control (QC) procedures for the student interview included frequent 
monitoring of telephone interviewers, a help desk that tracked and resolved difficulties encountered 
by sample members attempting to complete the web interview, and quality circle meetings and a 
debriefing for interviewers and tracers. Feedback from these procedures provided useful 
information for consideration when planning future administrations of BPS. 

Transcripts 

Postsecondary transcripts were collected as part of BPS:04/09. Transcripts were requested 
from all postsecondary institutions attended between July 1, 2003 and June 30, 2009 by members of 
the BPS:04 cohort. The BPS:04/ 09 transcript collection was combined with the transcript collection 
for the 2008/09 Baccalaureate and Beyond Longitudinal Study under the 2009 Postsecondary 
Education Transcript Study (PETS:09) to ease burden on institutions by sending institution staff a 
single request for transcripts which included sample members from both studies. 

Transcripts were requested from 3,030 eligible postsecondary institutions attended by sample 
members. This included institutions reported by sample members in the first follow-up (BPS:04/ 06) 
and second follow-up (BPS:04/ 09) student interviews as well as those identified on other transcripts 
received for sample members. Of the eligible institutions, 2,620 (87 percent) provided transcripts for 
the cohort. Many transcript submission methods were available to institutions, including several 
secure electronic methods, fax, and FedEx. Information and instmctions were available on a study 
website and institution contacting staff was also available to assist institution staff with transcript 
submissions and questions about the study. 

Transcript based data were created through a specially-designed keying and coding system 
(KCS) and a staff of trained keyer/ coders. The KCS was divided into sections based upon the 
categories of data found on transcripts, including case information, schools and terms, academics, 
tests, degrees and majors, and courses. A PETS coder was developed for the coding of courses by 
combining the 2010 NCES Classification of Instmctional Programs (CIP) and the 2003 College 
Course Map (CCM). 

Keying and coding was performed for 16,960 sample members (92 percent) on a total of 
25,120 transcripts. A series of quality control procedures were in place for keying and coding, 
including key-rekey and expert coder procedures to assess interrater reliability and upcoding 
procedures for uncodeable data and data entered as “other, specify.” Statistics were calculated to 
assess interrater reliability for multiple transcript data elements and all indicated substantial 
agreement between coders. 
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Executive Summary 


Analysis Weights 

Study respondents for BPS:04/ 09 were sample members with either BPS:04/ 09 interview 
data or enrollment data from administrative databases or transcript data. Weights were constmcted 
for use in the analysis of the three types of respondents: BPS:04/ 09 study respondents, panel 
respondents (i.e., study respondents to all three of NPSAS:04, BPS:04/06, and BPS:04/09), and 
BPS:04/ 09 sample members with transcript data. The three analysis weights were derived from the 
BPS:04/ 06 analysis weights; they were adjusted for nonresponse and calibrated to the sums of the 
BPS:04/ 06 weights. 

Variance Estimation 

Because the BPS:04/ 09 sample was obtained using a complex sample design that included 
stratification and clustering, special techniques were needed for variance estimates. Variables were 
constructed and provided on the data file for bootstrap variance estimation and for Taylor series 
variance estimation. 

Data Dissemination 

The data files for BPS:04/ 09 contain student-level data collected from student interviews 
and transcripts, government databases, and administrative databases. These files are available as a set 
of restricted research files fully documented by an electronic codebook (ECB) and through the 
National Center for Education Statistics’ online application PowerS tats, which also contains variable 
documentation. The editing and documentation processes for each file are described in this report. 
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Foreword 


This report describes and evaluates the methods and procedures used in the 2004/ 09 
Beginning Postsecondary Students Longitudinal Study (BPS:04/09). BPS:04/09 is the second and 
final follow-up interview for the cohort of first-time beginning postsecondary students identified in 
the 2004 National Postsecondary Student Aid Study. For the first time in BPS, in addition to the 
student interview, transcripts were collected from all of the postsecondary institutions attended by 
the sample. Together, the student interview and transcript data collections represent a significant and 
rich data source on this cohort of first-time beginning students. 

We hope that the information provided in this report will be useful to interested readers. 
Additional information about BPS:04/ 09 is available on the Web at 
http:/ /www.nces.ed.gov/ surveys /bps . 


Tom Weko 

Associate Commissioner 

Postsecondary, Adult, and Career Education Division 
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Chapter 1. 
Overview 


This report documents the methodological procedures and data quality evaluations of the 
2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). RTI International/ 
with the assistance of MPR Associates, Inc., conducted the BPS:04/09 study for the National Center 
for Education Statistics (NCES) of the U.S. Department of Education (under Contract No. ED-02- 
CO-OOll). 

Chapter 1 describes the background, legislative authorization, and schedule and products of 
BPS:04/ 09, as well as the major design changes from previous BPS studies. Chapter 2 presents the 
sampling details of the BPS:04 cohort and provides the definition of a BPS:04 cohort study 
respondent. Chapter 3 describes the development of the student interview and details of the data 
collection and results and provides an evaluation of the student interview data quality. Chapter 4 
describes the transcript data collection, including systems for collecting, recording, and evaluating 
transcript data. Chapter 5 summarizes the file preparation process for the BPS:04/ 09 student 
interview and transcript data collections. Finally, chapter 6 provides information pertaining to the 
weighting and variance estimation procedures for BPS:04/09. Materials used during the full-scale 
student interview and transcript data collection are appended to the report and cited in the text 
where appropriate. 

Throughout this document, reported numbers of sample institutions and students have been 
rounded to ensure the confidentiality of individual student data. As a result, row and column entries 
in tables may not sum to their respective totals, and reported percentages may differ somewhat from 
those that would result from these rounded numbers. 

1.1 Background and Objectives of BPS 

BPS is one of several NCES-sponsored studies developed to address the need for nationally 
representative data on key postsecondary education issues. BPS cohorts include first-time beginners 
(FTBs) in postsecondary schools who are surveyed at three points in time: in their first year in the 
National Postsecondary Student Aid Study (NPSAS), and then three and six years after first starting 
in postsecondary education in the BPS follow-up surveys. BPS collects data on a variety of topics, 
including student demographic characteristics, school and work experiences, persistence, transfer, 
and degree attainment. Previous cycles of NPSAS, as well as BPS, are authorized by the following 
legislation: 

• the Higher Education Act of 1965, as amended by the Higher Education Opportunity 
Act of 2008, 20 U.S.C. § 101 5a(k) (2010); 

• the General Education Provisions Act, as amended, 20 U.S.C. §§ 9541 to 9548 (2007); 

• the National Education Statistics Act of 1994, as amended, 20 U.S.C. §§ 9541 to 9547 
and 9573 (2007). 

Figure 1 shows the data collection timeline for the base-year and subsequent follow-up 
studies for each BPS study in the series. 


* RTI International is a trade name of Research Triangle Institute. 
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Figure 1. Chronology of BPS: 1990-2009 
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NOTE: NPSAS = National Postsecondary Student Aid Study. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


For the BPS:04 cohort, the first follow-up interview (BPS:04/ 06) captured the academic 
progress and persistence in postsecondary education of 2003-04 FTBs in the 3 academic years after 
their initial entry into a postsecondary institution. Data collected as part of this interview focused on 
continued education and experience, education financing, entry into the workforce, and the 
relationship between experiences during postsecondary education and various societal and personal 
outcomes. 

The second follow-up interview for the BPS:04 cohort, BPS:04/ 09, monitored students’ 
academic progress in the 6 academic years after their first entry into postsecondary education and 
assessed completion rates with a focus on the bachelor’s degree. Data collection continued to focus 
on education and employment, and the survey included many of the questions used in the first 
follow-up interview to provide continuity over time. BPS:04/ 09 also collected detailed information 
about the transition into employment after bachelor’s degree completion and asked questions related 
to family formation and personal change during this time. See section 3.1.1. Student Interview 
Design for a description of the contents of the BPS:04/ 09 student interview. 
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1 .2 Addition of T ranscript Component 

In addition to the student interview, BPS:04/ 09 included collection of postsecondary 
transcripts for the first time. Transcripts and course catalogs were requested from all institutions 
attended by the BPS:04 cohort since the first year of enrollment in 2003-04. Institutions were 
identified from interviews with the sample at the three time points — 2004, 2006, and 2009 — and 
from transcripts received from those institutions. Once received, the transcripts were keyed and 
coded using a data entry system specifically designed for postsecondary transcripts. Final raw data 
files and derived variables were included on the study’s electronic codebook (ECB), and made 
available through NCES’s online application PowerStats. Tables and regression analyses can be mn 
by any user through PowerStats, which also contains variable documentation. PowerStats is available 
online via the DataLab site at http:/ /nces.ed.gov/datalab/index.aspx . 

1.3 Schedule and Products of BPS:04/09 

Various activities comprised the student interview and transcript data collections. Table 1 
summarizes the schedule for BPS:04/ 09 student interview and transcript activities. 


Table 1. Schedule of major activities: 2007-10 


Activity 

Start date 

End date 

Student interview 

Finalize student sample 

8/1/2007 

9/30/2007 

Conduct web and telephone student interview data collection 

2/24/2009 

9/30/2009 

Conduct field student interview data collection 

6/22/2009 

9/30/2009 

Process student interview data; construct data files 

2/25/2009 

8/27/2010 

Transcript 

Collect postsecondary catalogs and transcripts 

8/1/2008 

3/5/2010 

Key and code transcripts 

11/2/2008 

5/7/2010 

Process transcript data 

2/2/2009 

8/13/2010 

Create transcript derived variables 

12/1/2009 

4/29/2011 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


In addition to this methodology report, BPS:04/ 09 includes several data products for the 
BPS:04 cohort which can be found on the NCES website at 

http:/ /nces.ed.gov/pubsearch/getpubcats.asp?sid=014 . A First Look report provides a brief 
description of persistence and degree attainment of the cohort over six academic years. Web tables 
provide 6-year attainment, persistence, transfer, retention, and withdrawal rates. A Statistical 
Analysis Report investigates the percentage of students in the cohort who began postsecondary 
education in degree programs but left without earning any credential, the timing of their departure, 
their personal and institutional characteristics, and the number of credits they earned prior to 
departure. Data files contain data collected from student interviews and transcripts, government 
databases, and administrative databases. These files are available as a set of restricted research files 
fully documented by an electronic codebook (ECB) and the public-use data are also available 
through the NCES online application PowerStats, which contains variable documentation and can 
be accessed at http:/ /nces.ed.gov/datalab/ . Table 2 provides release dates for BPS:04/09 data 
products. 
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Table 2. Data products: 2010-11 


BPS data products 

Release date 

First Look 

12/1/2010 

Powerstats (student interview) 

12/1/2010 

ECB (student interview) 

3/8/2011 

Web tables 

7/13/2011 

Methodology report 

9/30/2011 

Statistical Analysis Report 

12/31/2011 

Powerstats (transcripts) 

10/31/2011 

ECB (transcripts) 

11/30/2011 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 
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The BPS:04 cohort is a sample of first-time beginners (FTBs) first identified in the 2004 
National Postsecondary Student Aid Study (NPSAS:04) base-year study and surveyed again in the 
first follow-up study, the 2004/ 06 Beginning Postsecondary Students Longitudinal Study 
(BPS:04/ 06), and the second follow-up study, the 2004/ 09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). The BPS:04 cohort includes sample members that participated in 
one or more of the three interviews, as well as BPS:04 cohort study respondents for whom sources 
other than interviews provided critical information. 

2.1 Target Population 

The target population (or universe) for the BPS:04 cohort consisted of all students who 
began their postsecondary education for the first time during the 2003-04 academic year at any 
postsecondary institution in the United States or Puerto Rico that was eligible for NPSAS:04. 

2.1.1 Institution Universe for NPSAS:04 

The institutions eligible for NPSAS:04 were required during the 2003-04 academic year to 
meet all criteria for distributing federal aid authorized under Title IV of the Higher Education Act 
(20 U.S.C. §§ 1070-1099), including 

• offering an educational program designed for persons who have completed a high school 
education; 

• offering at least one academic, occupational, or vocational program of study lasting at 
least 3 months or 300 clock hours; 

• offering courses that are open to persons other than the employees or members of the 
company or group (e.g., union) that administers the institution; and 

• being located in the 50 states, the District of Columbia, or Puerto Rico. 

Institutions providing only vocational, recreational, or remedial courses or only in-house 
courses for their own employees were excluded. U.S. service academies were excluded because of 
their unique funding/ tuition base. 

These institution eligibility conditions are consistent with previous NPSAS studies, with two 
exceptions. First, the criterion of being eligible to distribute Title IV aid was implemented beginning 
with NPSAS^OOO, 1 and second, the previous NPSAS studies excluded institutions that offered only 
correspondence courses. NPSAS:04 included such institutions if they were eligible to distribute 
Title IV student aid. 

2.1.2 Student Universe for NPSAS:04 

Consistent with previous NPSAS studies, the students eligible for NPSAS:04 were those 
who were enrolled in eligible institutions and who satisfied both of the following eligibility 
requirements: 


1 An indicator of Title IV eligibility has been added to the analysis files from earlier NPSAS studies to facilitate comparable analyses. 
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• they were enrolled in either (1) an academic program, (2) at least one course for credit 
that could be applied toward fulfilling the requirements for an academic degree, or (3) an 
occupational or vocational program that required at least 3 months or 300 clock hours of 
instruction to receive a degree, certificate, or other formal award; and 

• they were not concurrently or solely enrolled in high school or in a General Educational 
Development (GED) program or other high school completion program. 

2.2 Base-Year Study (NPSAS:04) 

The sampling design for NPSAS:04, the base year study for BPS, was a two-stage design in 
which eligible institutions were selected in the first stage and eligible students, within eligible 
responding sample institutions, were selected in the second stage. The NPSAS:04 sampling process 
and first follow-up (BPS:04/ 06) subsampling procedures are described in the following subsection. 
For detailed information on the NPSAS:04 sample allocation and statistical design formulas, see 
appendix A. 

2.2.1 Institution Sample for NPSAS:04 

The institution sampling frame for NPSAS:04 was constmcted from the 2000-01 and 2001- 
02 Integrated Postsecondary Education Data System (IPEDS) Institutional Characteristics files and 
header files, and the 2000 and 2001 Fall Enrollment files. The sample of institutions was freshened 
using the 2002—03 IPEDS to include a sample of newly formed institutions. Records on the IPEDS 
files for NPSAS-ineligible institutions were deleted. NPSAS-ineligible institutions included U.S. 
service academies, institutions located outside the United States and Puerto Rico, and institutions 
offering no programs of study lasting at least 3 months or 300 clock hours. The IPEDS files were 
cleaned to resolve any of the following types of problems: 

• missing enrollment data, 2 because these data are needed to compute measures of size for 
sample selection; and 

• unusually large or small enrollment, especially if imputed, because, if incorrect, these data 
would result in inappropriate probabilities of selection and sample allocation. 

The institutions with cleaned data were then included in the sampling frame, and a direct, 
unclustered sample of institutions was selected for NPSAS:04. 3 As a part of this sample, to allow for 
analysis of the effects of state tuition and student aid policies in individual states, the number of 
institutions in 12 specific states was oversampled from three institution types — public 2-year 
institutions, public 4-year institutions, and private nonprofit 4-year institutions — in each of the 
following 12 states: California, Connecticut, Delaware, Georgia, Illinois, Indiana, Minnesota, 
Nebraska, New York, Oregon, Tennessee, and Texas. These 12 states were chosen from a set of 
volunteering states that expressed interest in and a willingness to support and encourage 
participation in NPSAS by their institutions. 


2 Missing IPEDS enrollment data had been previously imputed for most, but not all, of the NPSAS:04 institutions. This step filled in 
missing data for any remaining institutions. 

3 A direct unclustered sample of institutions was also selected in NPSAS:96. Prior to NPSAS:96, a clustered sample of institutions was 
selected for the study. 
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Table 3 presents the allocation of the NPSAS:04 institution sample by institution type. The 
number of sampled institutions was 1,670, of which 1,630 were eligible. 4 Of the 1,630 eligible 
institutions, 1,360 (84 percent) provided student enrollment lists. 

Table 3. NPSAS:04 institution sample sizes and yield, by institution type: 2004 


Eligible institutions that provided 
enrollment lists 


Institution type 

Sampled 

institutions 

Eligible 

institutions 

Number 

Unweighted 

percent 

Weighted 

percent 

Total 

1,670 

1,630 

1,360 

83.5 

80.0 

Public 

Less-than-2-year 

70 

60 

50 

76.6 

74.3 

2-year 

380 

380 

320 

85.4 

77.6 

4-year non-doctorate-granting 

130 

130 

110 

85.1 

70.3 

4-year doctorate-granting 

230 

230 

200 

86.3 

87.1 

Private nonprofit 

2-year or less 

70 

70 

70 

89.0 

92.6 

4-year non-doctorate-granting 

280 

270 

220 

81.9 

78.1 

4-year doctorate-granting 

220 

220 

170 

77.7 

80.8 

Private for-profit 

Less-than-2-year 

170 

160 

140 

84.0 

82.3 

2 years or more 

110 

110 

90 

84.4 

88.2 


NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003-04 National Postsecondary Student Aid 
Study (NPSAS:04). 


2.2.2 Student Sample for NPSAS:04 

The NPSAS:04 student sampling design was based on fixed-type sampling rates, rather than 
fixed-type sample sizes, to keep the probabilities of selection equal across student type within the 
institution type. The NPSAS:04 sampling design used 

• two student sampling types for undergraduates (one for FTBs, and one for all other 
undergraduates); 

• one type for first-professional students; and 

• three student sampling types for graduate students (master’s, doctoral, and “other” 
graduate students). 5 

The identification of an adequate number of FTBs for the NPSAS:04 sample was critical for 
preparing the first follow-up (BPS:04/06) sample. For the NPSAS:04 sampling frame, students who 
were thought to be FTBs based on an FTB indicator provided by sampled institutions on their 
student enrollment lists were classified as potential FTBs. NPSAS-eligible students who enrolled in a 
postsecondary institution during the NPSAS year (July 1, 2003, through June 30, 2004) for the first 


4 Among the ineligible institutions, approximately one-third closed after the sampling frame was defined, one-third failed to meet one 
or more of the NPSAS criteria for institution eligibility, and the remaining one-third had enrollment lists that were combined with 
eligible institutions because of an affiliation between the campuses. 

5 Differential sampling rates were used for the three types of graduate students to achieve adequate representation of students 
pursuing doctoral degrees and to limit the sample size for “other” graduate students, who are of limited inferential interest. 
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time after completing high school requirements were considered pure FTBs. Those NPSAS-eligible 
students who had enrolled for at least one postsecondary course before the 2003-04 NPSAS year 
but never completed that course were considered effective FTBs. Potential FTBs included both pure 
and effective FTBs. 

Samples from previous NPSAS studies provide evidence that postsecondary institutions are 
sometimes unable to accurately identify their FTBs. The false-positive and false-negative FTB rates 
experienced in NPSAS:96 were used to set appropriate FTB sampling rates for NPSAS:04. 6 

A total of 109,210 students were selected for the NPSAS:04 student sample from among the 
various institution types in the institution sample. The student sample included 49,410 potential 
FTBs, 47,680 non-FTB or “other” undergraduate students, and 12,120 graduate and first- 
professional students. 

Various sources were used to verify eligibility of the NPSAS:04 student sample, including an 
institution record abstraction (computer-assisted data entry [CADE]), the NPSAS:04 student 
interview, and record matching against several administrative databases (e.g., the U.S. Department of 
Education’s Central Processing System [CPS]). Of the 109,210 sampled students, 101,010 were 
found to be eligible for NPSAS:04. Ineligible students were identified during the student interview 
or from institutional records if student eligibility was not determined from a student interview. 

NPSAS:04 study respondents were those sample members for whom key pieces of data were 
obtained from one or more of the study sources. Students could be NPSAS:04 study respondents 
without completing the student instrument. Ninety percent of the eligible sample were considered 
study respondents. For more information about the NPSAS:04 study respondent definition and the 
NPSAS:04 sample, see the NPSAS:04 Full-Scale Methodology Report (Cominole et al., 2006). Table 
4 shows numbers of NPSAS:04 sampled and eligible students as well as response rates, by institution 
type and student type. 


6 The NPSAS:96 false-positive rate was 27.6 percent for students who were identified as potential FTBs by the sample institutions but 
later were determined not to be FTBs. The false-negative rate was 9.1 percent for those students who were not identified as potential 
FTBs by the sample institutions but later were determined to be FTBs. 
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Table 4. Numbers of NPSAS:04 sampled and eligible students and response rates, by institution 
type and student type: 2004 


Study respondents 1 


Institution type and student type 

Sampled 

students 

Eligible 

students 

Unweighted 

percent 

Weighted 

percent 

All students 

109,210 

101,010 

89.8 

91.0 

Institution type 

Public 

Less-than-2-year 

3,180 

2,580 

84.2 

90.6 

2-year 

36,300 

32,450 

81.3 

83.9 

4-year non-doctorate-granting 

9,200 

8,880 

91.9 

93.3 

4-year doctorate-granting 

22,350 

21,620 

93.7 

94.2 

Private nonprofit 

Less-than-4-year 

3,060 

2,770 

94.3 

94.6 

4-year non-doctorate-granting 

9,740 

9,300 

96.3 

96.9 

4-year doctorate-granting 

9,930 

9,590 

94.5 

95.4 

Private for-profit 

Less-than-2-year 

9,270 

8,030 

94.9 

94.3 

2 years or more 

6,190 

5,790 

95.0 

96.7 

Student type 

Total undergraduates 

97,090 

89,480 

89.3 

90.3 

Potential FTB 

49,410 

44,670 

91.2 

91.4 

Other undergraduates 

47,680 

44,810 

87.3 

90.0 

Graduate/first-professional 

12,120 

11,530 

94.2 

95.1 


1 A study respondent is defined as any eligible student for whom sufficient data were obtained from one or more 
sources, including student interview, institutional records, and other administrative data sources. 

NOTE: Detail may not sum to totals because of rounding. FTB = first-time beginner. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003-04 National Postsecondary 
Student Aid Study (NPSAS:04). 

2.3 First Follow-up Study (BPS:04/06) 

To construct the frame for the first follow-up (BPS:04/06) sample, multiple data sources 
containing information — such as a student’s year of high school graduation, undergraduate class 
level, and dates of receipt of any loans — were used to estimate or confirm a student’s likelihood of 
being an FTB during the NPSAS year. These data sources included 

• the base-year student interview (NPSAS:04); 

• student-level data obtained from institutional records via CADE; 

• the CPS, which contains data provided to the U.S. Department of Education by students 
and their families when they complete the Free Application for Federal Student Aid 
(FAFSA); and 

• the U.S. Department of Education’s National Student Loan Data System (NSLDS), 
which contains Pell Grant and Stafford Loan information. 

Using these indicators, a set of decision rules was developed to identify which cases had enough 
information confirming their FTB status to be included in the first follow-up sample. The first 
follow-up sample included 23,090 FTBs, identified as follows: 
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1. FTBs. Approximately 24,990 students responding to the base-year (NPSAS:04) student 
interview indicated that they were FTBs during the 2003-04 academic year. 
Approximately 3,820 of these students were identified for exclusion from the first 
follow-up (BPS:04/ 06) sample when the multiple data sources confirmed that they could 
not have been FTBs during the NPSAS year. Of the approximately 21,170 included in 
the first follow-up sample, approximately 19,800 had other data that strongly supported 
their FTB status, and approximately 1,370 of these students had some indications that 
they were not FTBs; these potential false positives were rescreened during the first follow- 
up interview to confirm their status. 

2. “Other” undergraduates (false-negative FTBs). Approximately 1,420 students were 
not originally classified as FTBs and were part of the base-year (NPSAS:04) group of 
28,610 “other” undergraduates but were later identified as potential FTBs based on 
either CPS data or because they had a high school graduation date in 2003 or 2004; these 
1,420 potential false negatives were included in the first follow-up (BPS:04/06) sample and 
rescreened during the first follow-up interview to verify their status. 

3. Study respondents likely FTBs. Approximately 8,860 students did not respond to the 
base-year (NPSAS:04) student interview but were classified as NPSAS:04 study 
respondents and were potential FTBs based on CADE, CPS, and loan data. Because 
student interview nonrespondents tend to have different demographic characteristics 
than interview respondents, approximately 460 of these 8,860 NPSAS:04 student 
interview nonrespondents were included in the first follow-up (BPS:04/ 06) sample to 
reduce nonresponse bias. Two factors, stratification by tracing outcome and the 
likelihood of being an FTB, were used to sample the 460 students most likely to be 
located and eligible for the study. 

4. Study nonrespondents likely FTBs. Approximately 720 base-year (NPSAS:04) sample 
members were potential FTBs based on information from CADE or CPS but did not 
respond to the base-year student interview and did not have sufficient data to be 
classified as study respondents. Of these 720 students, a subsample of approximately 40 
was included in the first follow-up (BPS:04/ 06) sample based on the same criteria 
(likelihood of eligibility and of being located) as the subsample in group 3. 

A visual representation of the distribution of the first follow-up (BPS:04/ 06) sample, by base-year 
response status, is shown in figure 2. 
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Figure 2. Distribution of BPS:04/06 sample, by base-year response status: 2004 
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NOTE: A study respondent is defined as any eligible student for whom sufficient data were obtained from one or more sources, 
including institutional records and other administrative data sources. Detail may not sum to totals because of rounding. FTB = first- 
time beginner. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003-04 National Postsecondary Student Aid 
Study (NPSAS:04) and 2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06). 


2.4 Second Follow-up Study (BPS:04/09) 

The first follow-up (BPS:04/06) starting sample consisted of 23,090 students. Over the 
course of the first follow-up data collection, 4,450 ineligible cases were removed from the sample 
based on the following: 

• responses to eligibility questions in the first follow-up student interview; 

• logistic modeling done using NSLDS and CPS data to predict the eligibility status of first 
follow-up interview nonrespondents; and 

• review of sample member eligibility information against updated National Student 
Clearinghouse (NSC) data. 

As a result of these procedures, the total second follow-up (BPS:04/ 09) sample was cleaned 
to remove the 4,450 ineligible cases and thereby reduced from 23,090 to 18,640 cases. Table 5 shows 
the distribution of the second follow-up sample by prior-round response status (i.e., whether the 
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student responded to the base year, NPSAS:04 interview, and the first follow-up, BPS:04/ 06 
interview). 


Table 5. BPS:04/09 sample size, by response status to prior-round interviews: 2009 


NPSAS:04 study 
respondent 1 

NPSAS:04 interview 
respondent 

BPS:04/06 interview 
respondent 

BPS:04/09 

sample 

Total 



18,640 

Yes 

Yes 

Yes 

14,750 

Yes 

Yes 

No 

3,510 

Yes 

No 

Yes 

140 

Yes 

No 

No 

220 

No 

No 

Yes 

10 

No 

i — , 

No 

No 

20 


sources, including institutional records and other administrative data sources. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003-04 National Postsecondary 
Student Aid Study (NPSAS:04), 2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06), and 
2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). 


Figure 3 shows the evolution of the BPS:04 cohort. 


Figure 3. Evolution of BPS:04 cohort: 2009 



NOTE: FTBs = first-time beginners. NPSAS:04 = 2003-04 National Postsecondary Student Aid Study (NPSAS:04). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

2.5 BPS:04 Cohort Study Respondent 

In addition to the BPS:04/ 09 student interview, student-level data for BPS:04/ 09 were 
collected from a variety of administrative sources, including the NSC StudentTracker and NSLDS 
files. Data from these other sources supplemented interview data and allowed enrollment histories 
and persistence and attainment variables to be constructed for a proportion of interview 
nonre spondents . 

A BPS:04/ 09 study respondent was defined as any sample member who was determined to 
be eligible for the study, was still alive at the time of the BPS:04/ 09 data collection, and had the 
requisite valid data from any source to allow construction of his or her enrollment history. 

Of the 18,640 cases included in the BPS:04/09 data collection, 110 were found to be 
deceased at the conclusion of data collection and removed from the sample, 15,160 were classified 
as study respondents on the basis of having completed a BPS:04/ 09 student interview, and 1 ,520 
were classified as BPS:04/ 09 study respondents because sufficient enrollment information about 
their postsecondary experience could be gathered from the NSC StudentTracker and NSLDS files. 
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Outcomes, and Evaluation 

The 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09), was 
designed for web, telephone, and field administration and included an abbreviated Spanish interview. 
Sample members were primarily located using batch address and phone sources and were asked to 
complete the student interview between February and October 2009. Analyses and evaluation of 
data collection provided information for consideration when planning future administrations of 
BPS. 

3.1 Student Interview Design and Systems 

The second follow-up interview (BPS:04/ 09) consisted of four sections. Enrollment History, 
Enrollment Characteristics, Employment, and Background, which were also in the first follow-up 
(BPS:04/ 06) student interview. Based on review of the relevance of specific interview topics, 
question wording and section structure were sometimes altered for the second follow-up interview. 
A Spanish interview in web mode was also added. This section provides the details of the second 
follow-up student interview design and systems. 

3.1.1 Student Interview Design 

The content of the second follow-up (BPS:04/ 09) interview remained largely the same as 
that in prior BPS second follow-up interviews (BPS:90/94 and BPS:96/01), building on data 
elements developed with input from the study’s Technical Review Panel (TRP) and from the 
National Center for Education Statistics (NCES).(For a list of TRP members, see appendix B.); As 
described in the BPS:04/09 Field Test Working Paper (Wine et al.), information learned in the field 
test also informed the full-scale student interview design. (For a list of the final set of student 
interview data elements, see appendix C.) 

The second follow-up interview consisted of four sections, grouped by topic. Respondents 
were guided through each section of the interview according to skip logic that took into account 
previously provided information from the base-year, 2003-04 National Postsecondary Student Aid 
Study (NPSAS:04), and the first follow-up study (BPS:04/06) and information recorded as the 
respondent progressed through the second follow-up (BPS:04/ 09) interview. Following are 
descriptions of the four interview sections. 

1 . Enrollment History collected information about all undergraduate enrollment from 
July 2006 through June 2009, with a focus on the attainment of a bachelor’s degree. It 
also captured any postsecondary enrollment following attainment of the bachelor’s 
degree. 

2. Enrollment Characteristics gathered information on the respondent’s experiences at 
the primary undergraduate school, which was the school where he or she had earned a 
bachelor’s degree or, if no bachelor’s degree had been earned, where he or she most 
recently had been enrolled. This section captured the respondent’s major or field of 
study, any employment while enrolled at the primary undergraduate school, and any 
undergraduate financial aid received. 
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3. Employment collected information about the respondent’s current employment status 
and job description (e.g., occupation, industry, earnings, satisfaction). This section 
applied to all bachelor’s degree recipients regardless of current enrollment status and to 
any other respondents who were not enrolled at the time of the interview. 

4. Background obtained information about student demographic characteristics, including 
citizenship, voting behavior, marital status and family composition, annual income and 
monthly expenses, volunteer activity, disability status, and education and teaching plans. 

The interview sections and principal topics in each section are summarized in figure 4. For 
the complete BPS:04/ 09 full-scale instmment facsimile, see appendix D. 

Figure 4. Interview sections and topics: 2009 

1 Enrollment History 2 Enrollment Characteristics 

Additional undergraduate schools attended Major or field of study 

since 2006 Grade point average 

Degree attainment since 2006 Job(s) while enrolled 

Any postbaccalaureate enrollment Undergraduate financial aid 

Satisfaction with choice of institution, major, 
and undergraduate education 

3 Employment 1 4 Background 

Occupation, employer type, industry Citizenship and voting 

Job earnings and benefits Marital status and household composition 

Relationship of job to undergraduate course of study Annual income and monthly expenses 

Licenses and professional certifications Volunteer activity 

Periods of unemployment Educational and teaching plans 

Disabilities 

1 Only bachelor’s degree recipients and respondents who were not currently enrolled received this section. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

A single instmment was developed to be administered in three modes: web, telephone, and 
field. For telephone and field interviews, the interviewer accessed the web instmment through RTFs 
computer-assisted telephone interviewing Case Management System (CATI-CMS). 7 An abbreviated 
version of the interview — containing only the first two sections, Enrollment Flistory and Enrollment 
Characteristics — was also provided to sample members as a final effort to gain primary enrollment 
information from sample members unwilling to participate in the full interview. 

To minimize mode effects, specific methodological features were incorporated into the 
instrument to provide web respondents with the assistance normally provided by a trained 
interviewer: 

• help text on every form to define key terms and clarify question intent; 

• pop-up messages to correct responses that were out of range or in an incorrect format; 

• conversion text to encourage responses to critical items when these items were left 
unanswered; and 


7 CATI-CMS is the system that assigns cases to be called and provides telephone interviewers with the appropriate screens and scripts 
to be used during the contacting and locating phase of CATI. 
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• pop-up messages prompting sample members to provide a response when they left three 
consecutive questions blank. 

Additionally, instmctions indicating how each question was to be administered (e.g., whether the 
response options were to be read aloud, when to probe) were included for telephone and field 
interviewers on each screen to minimize differences between interviews administered by an 
interviewer and web interviews. 

Coding systems. Assisted coding systems were used in the interview to standardize the 
collection and coding of the respondent’s postsecondary schools attended, major or field of study, 
and occupation. The name or title of each of these items was entered as a text string in each coder, 
and a keyword search conducted on an underlying database returned a list of possible matches. An 
assisted coding system was not used to code industries, but ready industry definitions allowed 
respondents and interviewers to select an industry classification from among a list of standardized 
options. Following are descriptions of the individual coding systems and sources: 

• The postsecondary school coder was developed from the set of institutions contained in the 
Integrated Postsecondary Education Data System (IPEDS), developed by NCES 
( http:/ /nces.ed.gov/ipeds A For any schools not listed in the database, respondents 
were asked to provide the control (e.g., public or private) and level (e.g., 4-year or 2-year) 
of the school. 

• The major coder was constructed using the 2010 Classification of Instructional Programs 
taxonomy, also developed by NCES (http:/ /nces.ed.gov/ipeds /cip20 10) . For any 
majors or fields of study not listed in the database, respondents were asked to provide a 
general major area and a specific discipline. 

• The occupation coderwas built from the Occupational Information Network Online 
(0*NET OnLine) database ( http:/ / online.onetcenter.org) . For any occupations not 
listed in the database, respondents were asked to provide a general area, a specific area, 
and finally a detailed classification area for the occupation. 

• The industry coder was based on the North American Industry Classification System 
(http:/ /www.census.gov/epcd/www/naics.html) . A text string was collected from the 
respondent, and then the respondent was asked to choose the category that best 
described his or her employer’s industry. Industry choices were laid out in general 
categories across the screen. When the respondent selected a category, examples of 
businesses within that industry were displayed, allowing the respondent to determine the 
appropriateness of the industry chosen. 

Spanish interview. A Spanish interview was developed from the BPS:04/ 09 abbreviated 
interview and for the first time was made available in web mode. This particular mode of 
administration required the translation into Spanish of not only question wording and response 
options but also of all the specific methodological features incorporated into the instmment to 
provide web respondents with the assistance normally provided by a trained bilingual interviewer 
(i.e., help text, pop-up messages to correct responses that were out of range or in an incorrect 
format, conversion text, and general error messages). The school names in the IPEDS coder and the 
majors in the major coders (the only coders in the abbreviated Spanish instmment) were not 
translated; however, instmctions were provided in Spanish to both respondents and bilingual 
telephone and field interviewers explaining that they should choose a school or major code in 
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English, if possible, or instead enter a text string in Spanish and not attempt the coding of the 
school or major. 

3.1.2 Data Collection Systems 

This section describes the data collection systems used for the BPS:04/ 09 data collection, 
including the Hatteras Survey Engine and Survey Editor (RTFs proprietary web-based interviewing 
software), the Instrument Development and Documentation System (IDADS), and the Integrated 
Management System (IMS). 

Hatteras Survey Engine and Survey Editor. The BPS:04/ 09 survey instmment was 
created with Hatteras, a web-based system in which project staff developed, reviewed, tested, 
modified, and communicated changes to specifications and code for the instmment. All information 
relating to the instmment was stored in an SQL Server database and was made accessible through 
web browser interfaces. Hatteras provided specification, programming, and testing interfaces for the 
BPS instmment as follows. 

• Specifications. Hatteras provided the tools and user interface for developing interview 
specifications. Specification content included wording at the form, question, item, and 
response option levels; help text content; item-level data documentation; and form-level 
question administration documentation. Specific capabilities of the Hatteras system 
allowed instmment designers to import any relevant specifications used in prior studies, 
create skip logic and item documentation, and search a library of survey items. 
Instrument designers were also able to take advantage of a comprehensive comment 
tracking system to communicate and test necessary instrument changes with 
programmers. 

A web interface provided access for project staff at MPR Associates, Inc. (MPR), and at 
NCES to test and comment on the instrument throughout development. 

• Programming code. For simple instrument questions and items, Hatteras automatically 
translated specifications into web page scripts when the web page was accessed. For 
questions involving complex routing, multiple question wording or response option 
conditions, or nonstandard page layout or behavior, programmers entered custom 
programming code — HTML, JavaScript, and C# .NET script — into the Hatteras custom 
code interface. This code was stored in the SQL database server, together with the 
instrument specifications for compilation by the survey execution engine. 

• Instmment testing and execution. The Hatteras system’s survey execution engine allowed 
immediate testing of specification and code content as it was developed and updated by 
displaying web content as respondents would see it. The execution engine also 
automatically handled such web instmment functions as backing up and moving 
forward, recording instmment timing data, displaying critical-item wording, validating 
user input, displaying conditional instructions based on interview mode (web, telephone, 
or field) and linking to context-specific help text. 

• Survey sites and data transfer- — Web/ telephone. For web and telephone data collection, the 
Hatteras survey execution system was installed on the surveys server and SQL database 
server at NCES. Web respondents accessed the survey directly by web browser after 
logging in with a user ID and password. RTFs telephone interviewers accessed the same 
NCES web survey site by means of a web browser process launched from RTFs CATI- 
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CMS. All connections to the NCES web interview were secured with Secure Sockets 
Layer (SSL) encryption. Automated processes transferred data between RTFs local 
database and the NCES database via a secure, encrypted connection. 

• Survey sites and data transfer — Field. For field interviews, the Hatteras survey execution 
system was installed on local web and database servers on laptop computers. Field 
interviewers accessed the laptop-based survey by logging in through three independent 
levels of security, including a whole-disk encryption outer level. Interview control and 
response data were transferred between RTI and field laptops via secure, encrypted, 
automated connections. 

Instrument Development and Documentation Systems (IDADS). The web-based 
IDADS documentation module contained the finalized version of all instmment items, their screen 
wording, and variable and value labels. Also included were the more technical descriptions of items 
such as variable types (alpha or numeric), to whom the item was administered, and frequency 
distributions for response categories based on completed interview data. The documentation 
module was used to generate the instrument facsimiles and the deliverable electronic codebook 
(ECB) input files. 

Integrated Management System (IMS). All aspects of the study were controlled using an 
IMS, a comprehensive set of desktop tools designed to give project staff and NCES access to a 
centralized, easily accessible repository for project data and documents. The BPS:04/09 IMS 
consisted of several components: the management module, the Receipt Control System (RCS) 
module, and the instrumentation module. 

• Management module. The management module of the IMS included tools and information 
to assist project staff and the NCES project officer in managing data collection. All 
management information pertinent to the study was located there, accessible via the 
Web, and protected by SSL encryption and a password-protected login. The IMS 
contained the current project schedule, monthly progress reports, daily data collection 
reports and status reports (generated by the RCS described below), project plans and 
specifications, project deliverables, instmment specifications, a link to the Hatteras 
system, staff contacts, the project bibliography, and a document archive. 

• Receipt Control System (RCS). The RCS is an integrated set of systems that was used to 
control and monitor all activities related to data collection, including tracing and locating. 
Through the RCS, project staff were able to perform tracing and data management 
operations, track case statuses, identify problems early, and implement solutions 
effectively. The RCS’s locator data were used for a number of daily tasks related to 
sample maintenance. Specifically, mailout systems produced paper mailings and 
e-mailings to sample members, the query system enabled administrators to review the 
locator information and status for a particular case, and the mail return system enabled 
project staff to update the locator database as mailings or address update sheets were 
returned or forwarding information was received. The RCS also interacted with the 
computer-assisted telephone interviewing (CATI) system, sending locator data between 
the two systems, as necessary. 

• Instrumentation module. The instmmentation module managed development of the 
multimode web data collection instrument within Hatteras. Developing the instrument 
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with Hatteras ensured that all variables were linked to their item and screen wordings 
and were thoroughly documented. 

3.2 Student Interview Data Collection 

The BPS:04/ 09 interview data collection involved training data collection staff and locating, 
contacting, and interviewing sample members. Each of these procedures is detailed in this section. 

3.2.1 Training of Interview Data Collection Staff 

Members of the data collection staff included quality control supervisors (QCS), help desk 
agents (HDAs), telephone interviewers, field interviewers, and intensive-tracing staff. Prior to 
beginning work on BPS, all data collection staff completed a comprehensive training program. 
Topics covered in training programs included a review of confidentiality requirements, an overview 
of the BPS:04/ 09 study, frequently asked questions (FAQs), and administrative procedures for case 
management as well as hands-on practice. All training programs were designed to maximize active 
participation of the trainees. The training schedule and number of data collection staff trained are 
presented in table 6. 


Table 6. Training of interview data collection staff: 2009 


Staff trained 

Time period 

Number of 
staff trained 

Quality control supervisors 

February 14, 2009, and March 12, 2009 

15 

Help desk agents 

February 16-19, 2009 

13 

Telephone interviewers 

March 17-19, 2009 

31 

Intensive-tracing staff 

April 1, 2009, and April 29, 2009 

16 

Field interviewers 

June 15-18, 2009, and July 27-31 , 2009 

43 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


The specific roles and duties of data collection staff are summarized in the following subsections, 
along with a description of the training program (see appendix E for training materials). 

Quality control supervisors. QCS provided support and guidance for the telephone 
interviewers, monitored interviewer production, and helped troubleshoot problems. They attended 
BPS project supervisor training and also participated in telephone interviewer project training. 
Training included an overview of BPS:04/09, conversational interviewing techniques expected of 
interviewing staff, problem resolution, case review, an explanation of project-specific reports, and 
other specific project procedures and protocols. The QCS were also provided with a supervisor 
manual to be used as a reference throughout the course of data collection. 

Help desk agents. A staff of help desk agents assisted any sample members who had 
questions or problems while completing web interviews. HDAs were certified telephone 
interviewers specially trained to unlock cases, reissue passwords, record and track calls to the study 
help line via the help desk application, and effectively respond to any caller’s questions. During the 
early response period, HDAs also made prompting calls to first follow-up (BPS:04/ 06) interview 
nonrespondents and completed telephone interviews with sample members who preferred a 
telephone to a web interview. Help desk training materials included a project telephone interviewer 
manual with a help desk supplement and various project handouts. 
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Telephone interviewers. Telephone interviewers were responsible for gaining cooperation 
from and conducting interviews with sample members, avoiding interview refusals, and addressing 
the concerns of reluctant sample members. Telephone interviewers received 16 hours of training 
that included an overview of the study, an in-depth review of the interview instmment, hands-on 
practice administering the telephone interview, review of appropriate conversational interviewing 
techniques, and practice with the CATI-CMS. At the conclusion of training, all telephone 
interviewers were certified by successfully conducting mock telephone interviews and by providing 
satisfactory responses to the study’s FAQs. Telephone interviewer training materials included a 
telephone interview manual and multiple project handouts. 

Field interviewers. Field interviewers conducted interviews, either in person or by 
telephone, with sample members residing in 25 selected geographic clusters in the United States and 
Puerto Rico. Field interviewers were required to attend a 3-day training session held in Durham, 
North Carolina. Prior to the classroom training, each field interviewer was required to complete a 
home study exercise. Field interviewers received classroom training similar to that of telephone 
interviewers, with additional training on the field CMS, field locating strategies, management of the 
case assignment folders, and proper care and use of the BPS laptops. Field interviewers were also 
required to conduct successful certification mock interviews and multiple other certification 
exercises before they were permitted to begin work. Field interviewer training materials included a 
field interviewer manual and additional handouts and forms documenting all field procedures and 
expectations of work. 

Tracing staff. Tracing staff (tracers) used intensive measures (described in section 3.2.3) to 
locate sample members designated as lacking good telephone contacting information. Tracers 
attended a comprehensive 16-hour training session led by RTI tracing managers and covering all 
tracing procedures. Tracers also received 2 hours of project-specific training. They received an 
overview of BPS, a review of the FAQs, background information on the BPS sample, and the 
tracing techniques best suited to locating BPS sample members. 

Additional trainings. Selected staff received additional training modules, such as refusal- 
conversion training, and Spanish interview training (certified bilingual staff). Additionally, quality 
circle meetings were routinely conducted as an extension of the training program for continual 
quality improvement. Data collection staff were given the opportunity to ask questions in quality 
circle meetings, and as needs were identified, additional training topics were highlighted and 
addressed in subsequent meetings. After each meeting, quality circle notes were posted on the call 
center’s project website and on the project IMS. 

3.2.2 Study Website 

BPS:04/ 09 sample members were provided with a link to the BPS website prior to the start 
of data collection. The website provided general information about the BPS set of studies, including 
details about the study sponsor and contractors, how the data are used, answers to FAQs, 
confidentiality assurances, and selected findings from earlier BPS studies. The website also provided 
contact information for the study help desk and project staff at RTI, as well as links to the main 
NCES and RTI websites. Sample members were able to log in to the secure portion of the website 
to provide updated contact information and complete the student interview once it became 
available. 

Designed according to NCES web policies, the BPS:04/09 study website used a three-tier 
security approach to protect all data collected. The first tier of security included secure log-ins, with 
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a unique study ID and strong password provided to sample members prior to the start of data 
collection. The second tier of security protected any data entered on the website with SSL 
technology, allowing only encrypted data to be transmitted over the Internet. The third tier of 
security stored any collected data in a secured SQL Server database located on a server machine that 
was physically separate from the web server. Figure 5 shows the home page for the BPS:04/ 09 study 
website. 


Figure 5. BPS:04/09 website home page: 2009 



condary Students 
tudy 


Sponsored By: National Center for Education Statistics, U.S. Department of Education 


Home 


Home 


About BPS 


FAQs 


OMB Clearance No.: 1850-0631 
Expiration Date: 12/31/2010 

Burden Statement 


Confidentiality 


Contact Us 


Login to the Interview 


Welcome to the Beginning Postsecondary Students Longitudinal Study (BPS) Website! The BPS study collects 
data related to enrollment in and transition out of postsecondary education. You were selected to participate in BPS 
because you first enrolled in college or other postsecondary education during the 2003-2004 school year. Study 
participants were first contacted in 2004 and again in 2006. Now we want to contact you one last time to find out 
about your education, employment, and other experiences during the past few years. 


To access the BPS interview online, enter your BPS study ID number and password, which are printed on the letter 
that was recently mailed to you, or included in an e-mail message you may have received. We recommend that you 
use Internet Explorer, Firefox, or Netscape as your browser to complete the web version, and you will need 
to temporarily disable any pop-up blockers you have enabled. For instructions on how to temporarily disable 
your pop-up blocker, click here . We thank you in advance for your participation. 


BPS study ID Number: | 

Password: | (the password is case-sensitive) 

Login | Forgot Password [ 


If you'd prefer to be contacted by a trained interviewer from RTI International, please click here . 

You can obtain additional information about the BPS study by using the links to the left of this page. If you need 
additional assistance, send an e-mail to bps@rti.org or call toll-free at 1-800-334-2321. 



National Center for Education Statistics, Institute of Education Sciences 

U.S. Department of Education 

1990 K Street, NW, Washington, DC 20006 USA 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

3.2.3 Locating and Contacting Sample Members 

Several locating methods were used to find and collect up-to-date contact information for 
the BPS:04/09 sample (figure 6). Batch searches of national databases and prenotification address 
update mailings were conducted prior to the start of data collection. After the start of data collection 
and for those sample members not yet found, follow-up locating methods were employed, including 
CATI locating, intensive tracing, and field tracing. 
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Figure 6. Locating methods: 2009 



NOTE: CATI = computer-assisted telephone interviewing. CPS = Central Processing System. NCOA = National Change of Address. 

SM = sample member. NR = nonresponse. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 

Longitudinal Study (BPS:04/09). 

• Batch tracing. Before mailing activities began, batch database searches were conducted 
to update sample member contact information. These searches used the U.S. 

Department of Education’s Central Processing System (CPS) and the U.S. Postal Service 
(USPS) National Change of Address (NCOA) databases. The information obtained from 
these sources was compared with the information previously available from the first 
follow-up (BPS:04/ 06) locator database to identify any new contact information. Then, 
just prior to the start of outbound telephone interviewing, all sample member addresses 
and telephone numbers were sent to Telematch, a computerized residential telephone 
number service with the not-yet-published numbers of new movers, to obtain any 
telephone number updates. 

• Mailings. To maintain contact with the BPS:04 cohort between the end of the first 
follow-up (BPS:04/ 06) data collection and the beginning of the second follow-up 
(BPS:04/09) data collection, a panel maintenance postcard was sent in November 2007. 
The postcard, signed by the BPS project director, asked sample members to visit the 
BPS student website to provide updated locating information. Any locating information 
collected from the panel maintenance update was combined with locating information 
from the first follow-up study and was used to populate the second follow-up study 
locator database. 

In January 2009, about 6 weeks before the start of BPS:04/09 data collection, a mailing 
went to the parents of sample members younger than 26 years old so as to gain their 
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assistance with providing up-to-date contact information for these sample members. 

This mailing included a study brochure, a letter with detailed information about 
BPS:04/09 signed by the associate commissioner of NCES, an address update sheet, and 
a business reply envelope. 

The final step in the pre-data collection locating and contacting effort occurred 
approximately 3 weeks before the start of data collection, with a similar address update 
mailing going to sample members (using any updated contact information provided by 
parents, if applicable). The mailing contained a letter notifying sample members of the 
upcoming BPS:04/ 09 data collection, a copy of the study brochure with responses to the 
FAQs, an address update sheet, and a business reply envelope. Sample members were 
asked to update their address information on the address update sheet and return it in 
the postage-paid envelope. They also had the option of entering the information using 
the online form available on the BPS website. The address update sheet and online form 
included a prompt for sample members to indicate a preference for being notified by 
text message at the start of data collection. 

A data collection announcement was mailed on February 23, 2009, to first follow-up 
(BPS:04/06) interview respondents in a 9 x 12 inch BPS envelope by USPS first-class 
mail and to first follow-up interview nonrespondents by USPS Priority Mail. The mailing 
to all sample members included a study brochure and a letter that announced the start of 
data collection (appendix F). The letter, signed by both the BPS project director and the 
NCES project officer, informed sample members of the cash incentive for completing 
the interview by March 23, 2009, provided the study website and sample member’s user 
ID and password for accessing the web interview, and provided the study’s toll-free help 
desk number and e-mail address. On February 25, 2009, an e-mail containing 
information comparable to that in the data collection announcement letter was sent to 
sample members. On February 26, 2009, a letter was mailed to parents of all sample 
members younger than 26 years old (when a parent address was available), explaining the 
importance of the study and asking parents to encourage sample members to participate. 
Additional mailings included a postcard reminder sent about 10 days after the data 
collection announcement and two additional e-mail reminders to encourage early 
interview response. Once outbound telephone interview efforts began and throughout 
data collection, periodic mailings and e-mails went to interview nonrespondents 
throughout the course of data collection. 

• CATI locating and pre-intensive tracing. Telephone interviewers made prompting 
calls to first follow-up (BPS:04/ 06) interview nonrespondents during the early response 
period of data collection. These calls, described in more detail in section 3.2.4, helped 
identify cases that required further tracing. Once outbound telephone interviewing 
began, telephone interviewers conducted limited tracing and locating activities, as 
needed. The telephone number believed to be the best known number for contacting the 
sample member was attempted first. If the sample member could not be reached at that 
number after several attempts, any other numbers associated with the sample member, 
including parent and other contacts, were called. If the sample member could not be 
located the case was designated for FastData and Accurint batch services which 
provided an automated search for matching phone numbers to sample members using 
combinations of address, name, and SSN. Cases for which neither FastData nor 
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Accurint batch generated new telephone numbers were sent for intensive interactive 
tracing by RTFs Tracing Operations (TOPS). 

Overall for BPS:04/ 09 data collection, the batch matching successfully confirmed 
contact information or provided new contact information for 29,370 records. 8 The most 
records, 11,050, were matched through Telematch. While the fewest records, 1,310, were 
matched through FastData, this data source minimized the number of cases requiring 
more costly intensive tracing. Table 7 shows the match rates for each tracing source. 


Table 7. Batch processing record match rates, by tracing source: 2009 


Tracing source 

Number of 
records sent 

Number of 
records matched 

Percent 

matched 

Total 

57,600 

29,370 

51.0 

CPS 

18,030 

7,500 

41.6 

NCOA 

18,530 

6,970 

37.6 

Telematch 

14,600 

11,050 

75.7 

FastData 

2,730 

1,310 

47.8 

Accurint 

3,700 

2,540 

68.7 


NOTE: Detail may not sum to totals because of rounding. CPS = Central Processing System. NCOA = National 
Change of Address. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 

• Intensive tracing. The most difficult locating cases were traced at TOPS using a two- 
tiered strategy and a number of sources. TOPS-1, the first tier, identified sample 
members with Social Security numbers (SSNs) to trace them through consumer 
databases (FastData’s SSN search and Experian) that contain current address and 
telephone listings for the majority of consumers with credit histories. If a search 
generated a new telephone number for the sample member, tracers attempted to confirm 
the information by speaking with the sample member or with someone else who could 
confirm the information. If the number was confirmed, the case was sent back to CATI 
for telephone interviewing. This first level of effort minimized the time that cases were 
in tracing and unavailable for CATI efforts. Cases still not located and not in a field 
cluster underwent a more intensive level of tracing in TOPS-2. TOPS-2 included calls to 
other possible sources of information, including, for example, directory assistance, 
alumni offices, and contacts with neighbors or landlords. Whenever any of these sources 
provided information that indicated a sample member was not available for the study 
(e.g., deceased, incarcerated, or out of the country), no further contact efforts were 
made. 

Prior to the start of the student interview data collection, 30 cases of the 18,640 sample 
were found to be deceased; overall, about 17 percent of the remaining 18,610 sample 
members required intensive tracing (table 8). Forty-three percent of the first follow-up 
(BPS:04/ 06) interview nonrespondents required intensive tracing, compared with 
11 percent of first follow-up interview respondents (/(4,458) = 37.85, A < .001 ). Thirty- 
four percent of sample members whose NPSAS institutions were private for-profit less- 
than-2-year schools required intensive tracing, compared with 8 percent of sample 


8 The number of records matched exceeds the number of sample members because individual sample members may have had their 
contact information sent to a number of the batch tracing sources. 
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members whose NPSAS institutions were private, nonprofit 4-year doctorate-granting 
schools (7(2,477) = 18.95,_/> < .001) or public, 4-year doctorate -granting schools 
«2,108) = 19.97, p < .001). 

Table 8. Cases requiring intensive tracing, by first follow-up response status and institution type: 
2009 

First follow-up response status and institution Cases requiring intensive tracing 


type Total Number Percent 


Total 

18,610 

3,180 

17.1 

First follow-up response status 

Respondent 

14,900 

1,590 

10.6 

Nonrespondent 

3,720 

1,590 

42.8 

Institution type 

Public 

Less-than-2-year 

600 

140 

24.0 

2-year 

6,380 

1,280 

20.1 

4-year non-doctorate-granting 

1,600 

170 

10.6 

4-year doctorate-granting 

3,080 

250 

8.0 

Private nonprofit 

2-year-or-less 

550 

130 

23.7 

4-year non-doctorate-granting 

2,210 

220 

9.9 

4-year doctorate-granting 

1,580 

120 

7.8 

Private for-profit 

Less-than-2-year 

1,570 

530 

33.8 

2 years or more 

1,050 

330 

31.6 


NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


• Field tracing. Any cases not located after TOPS-1 intensive tracing and thought to be 
in one of the 25 selected geographic field clusters were designated for field tracing 
instead of being sent to TOPS-2. Information provided to field interviewers included all 
address information available for an assigned case, the results of TOPS-1 intensive 
tracing efforts, and the details of all call attempts made by telephone interviewers. In 
addition to these tracing resources, field interviewers had access to contacts within the 
community, such as post office mail carriers or local public records that could provide 
additional information. Many field interviewers also had the added advantage of calling 
from telephones with local area codes familiar to sample members, increasing the 
likelihood that sample members would respond to the telephone calls. 

3.2.4 Interviewing 

Data collection for the BPS:04/09 interview consisted of three phases (figure 7): 

1. Early response phase. This phase began with the start of data collection in February 2009 
and lasted approximately 4 weeks, until March 23, 2009. First follow-up (BPS:04/06) 
student interview respondents who completed the second follow-up (BPS:04/ 09) 
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student interview during this phase received an incentive of $30; First follow-up student 
interview nonrespondents received a $50 incentive. 9 

2. Production phase. During this phase, interviewers called to encourage sample members to 
complete the interview by telephone or on the Web. First follow-up student interview 
respondents received a $20 incentive during this phase; First follow-up student interview 
nonrespondents received a $40 incentive. 

3. Nonresponse conversion phase. Cases in this phase belonged to one of the following groups: 
interview refusal by the sample member or a sample member contact, hard-to-reach , 10 not 
locatable after intensive tracing, insist-paj, n and field cases that did not fit into one of the 
preceding groups. First follow-up student interview respondents who completed the 
second follow-up interview during the nonresponse conversion phase were offered a $30 
incentive, and first follow-up student interview nonrespondents were offered a $50 
incentive. Data collection ended in October 2009. 


Figure 7. Data collection phases: 2009 


Early response phase 


Production phase 


Nonresponse conversion 
phase 

• First 4 weeks of data collection 

• Web interviews encouraged 

• $30 or $50 incentive 

* 

• Outbound telephone calls: web 
remains available 

• $20 or $40 incentive 

- 

• Sample members who refused, 
were hard to reach, or insisted on 
S30/$50 incentive 

• Outbound telephone calls: web 
remains available 

• $30 or $50 incentive 

W 

W 


✓ 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


Sample members could complete the interview on the Web or by telephone throughout the 
data collection period. The interview screens in the telephone and field interviews were identical to 
those in the web interviews, except that interviewer instructions on how to administer each question 
were visible at the top of each screen for telephone and field interviews. Following are details of the 
administration of the interview through the various modes. 

Web interviews. Sample members were informed of the web interview in the data 
collection announcement mailing. During the early response period (the first 4 weeks of data 
collection), only web interviews were completed unless sample members initiated a telephone 
interview by calling the help desk or sending an e-mail asking to be called. Reminder mailings and 
e-mails were sent throughout the production and nonresponse conversion phases of data collection 
to encourage sample members to complete the interview online. The website was accessible 
24 hours a day, 7 days a week, throughout the data collection period, providing sample members 
with the option to complete the interview online at any time. 

Help desk operations. The help desk for BPS:04/ 09 opened on February 24, 2009, in 
anticipation of the first respondent calls after the data collection announcement mailing. Flelp desk 


9 First follow-up (BPS:04/06) student interview nonrespondents received telephone prompting calls during the early response phase 
to remind them of their inclusion in the study and to encourage their participation. 

10 Hard-to-reach cases were those that were called at least 15 times (8 times for first follow-up nonrespondents) and yielded minimal or 
no contact with the sample member. 

11 Insist-pay cases were those in which sample members completed the interview during the production phase but insisted on receiving 
the incentive amount offered during the early response phase. 
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staff were available to assist sample members who had questions or problems accessing and 
completing the web interview. A toll-free help line was established to accept incoming help desk 
calls. If technical difficulties prevented sample members from completing the web interview, 

HDAs — also trained to conduct telephone interviews — would encourage sample members to 
complete a telephone interview. 

A help desk application was created to document incoming calls from sample members and 
other contacts. Specifically, the help desk application included the following: 

• information needed to verify the sample member’s identity; 

• login information needed by the sample member to access the web interview; 

• a means to update sample member contact information, as needed; 

• functionality to unlock cases and send an e-mail containing the website and study login 
information to the sample member; 

• systematic documentation of each call; and 

• a means for tracking calls that could not be resolved immediately. 

The help desk application provided project staff with a means to monitor the resolution 
status of all help desk events and reports on the type and frequency of problems experienced by 
sample members. 

Telephone interviews. Telephone follow-up locating and interviewing began on March 24, 
2009, after the 4- week early response period ended. Telephone interviewing procedures included 
attempts to locate, gain cooperation from, and interview sample members who had not yet 
completed the interview. Interviewers encouraged sample members to complete the interview by 
telephone; however, sample members could still complete the interview on the Web, if that was their 
preference. Sample members who did express a preference to complete a web interview were called 
back 5 days later for follow-up if the interview had not yet been completed. 

The CATI-CMS included an automated call scheduler that assigned cases to interviewers by 
case priority, time of day, day of week, existence of previously scheduled appointments, and type of 
case. Case assignment was designed to maximize the likelihood of contacting and interviewing 
sample members, and cases were assigned to various queues accordingly. For example, the CMS 
included queues for new cases that had not been called, Spanish-language cases, initial refusals, and 
various appointment queues. In addition, available telephone numbers for each case were 
automatically prioritized for the interviewers. As new roster lines were added — as a result of CATI 
tracing, other tracing efforts, and information from other sources such as respondent e-mails or help 
desk call-ins — available telephone numbers were reprioritized based on the new information. 

Some cases required special treatment. For cases with sample members or contacts who 
spoke only Spanish, bilingual interviewers were available to administer a Spanish interview (see 
section 3.1.1 for details regarding the Spanish interview). To gain cooperation from those sample 
members who initially refused to participate (as well as from contacts such as parents and 
roommates who acted as gatekeepers to the sample member), interviewers were trained in refusal- 
conversion techniques. As the end of data collection approached, all telephone interviewers were 
trained to administer the abbreviated English-language interview to any reluctant sample members. 
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Field interviews. Field data collection activities began approximately 3 months after the 
start of outbound telephone interviewing, during the nonresponse conversion phase of data 
collection. Using the last known address for each case, RTFs Geographic Information System (GIS) 
program conducted an analysis of the BPS:04/ 09 sample to identify the 25 geographic areas with the 
highest density of sample members residing within a 1 00-mile radius of the cluster center. On the 
basis of this analysis, 43 field interviewers were hired to work nonrespondent cases that required 
field efforts. An Integrated Field Management System provided reports that helped project staff 
manage the progress of the field interviewing effort. Once assigned to the field, cases were excluded 
from further outbound efforts from the call center but could still be completed on the web or by 
telephone if sample members called the help desk to complete the interview. See section 3.4.1 
Student Interview Response Rates for results of field, telephone, and web interviews. 

3.2.5 Other Procedures to Maximize Locating and Interview Response 

Throughout data collection, the BPS project team continued to work with TOPS and other 
available resources to evaluate additional tracing efforts that could benefit BPS data collection. In 
addition to the locating sources and methods already described, BPS:04/ 09 used several other 
procedures to maximize locating and interview response. 

Other locating methods. Other locating methods used to find sample members included 
the following: 

• Experian credit header search. Halfway through data collection, an Experian credit header 
search was conducted to obtain phone numbers and addresses associated with sample 
members according to their credit histories. This search provided a relatively low-cost 
alternative to other intensive tracing methods. Priority was given to sample members 
who had not yet been located or dead-ended in CATI, but the search was also conducted 
for sample members who were once located but never reached or who were located but 
not reached for several weeks. 

• FasfData ’s Superphone search. Halfway through data collection, TOPS also began 
conducting a FastData Superphone search for telephone numbers on cases that had 
recently gone through the first round of TOPS-1 tracing but for which no locating 
information was found. Unlike many other tracing sources, FastData’s new Superphone 
search provided cell phone numbers as well as landline numbers. 

• Experian MetroNet hatch search. During the nonresponse conversion phase of data 
collection, this search was conducted to find new contact information for all cases where 
interviews had not yet been completed. Experian MetroNet batch searches self-reported 
consumer-contacting databases. This search provides up-to-date contact information for 
more than 140 million households and is updated every 2 weeks. 

Other contacting methods. Text messaging and social networking (Facebook and 
MySpace) were additional methods used to contact sample members and encourage interview 
completion. 

• Text messages. Some sample members were contacted by Short Message Service 
technology, or text messaging. A text message reminder to complete the BPS:04/ 09 
interview was sent during the early response data collection period to those sample 
members who had requested on their address update sheet that a text message be sent. A 
text message reminder was also sent during the nonresponse conversion phase to sample 


BPS:04/09 Full-scale Methodology Report 


27 



Chapter 3. Student Interview Design , Data Collection , Outcomes , and Evaluation 


members with a cell phone number on record who had not yet completed the interview. 
Each text was sent manually from a project mobile phone. The text message included 
the sample member’s first name, mention of the BPS interview, expected length of the 
interview, the incentive amount available, and the name of the project staff person 
sending the text message. 

• Social networking. Two popular social networking sites, Facebook and MySpace, were used 
to generate new leads for and to make contact with sample members who were difficult 
to locate. BPS:04 cohort information on record — such as postsecondary institutions 
attended, city/ state networks, e-mail addresses, and birth dates — were used to search for 
sample members on Facebook or MySpace. Once the targeted individual was believed to 
be found, an e-mail message describing BPS and the incentive being offered, as well as 
reminding the individual of any past participation in the study, was sent through internal 
messaging on the social networking site. Although the message included the study 
website and help desk telephone number, no personally identifying information (such as 
login information) was included; this information could only be provided to a sample 
member who visited the study website or contacted the help desk and verified his or her 
identity. Efforts to contact sample members by Facebook were ceased because of 
restrictions on the number of messages sent to individuals. When few BPS messages 
were opened by sample members contacted through MySpace, the use of social 
networking sites was abandoned in favor of the other, more promising locating and 
contacting methods. 

3.3 Data Collection Quality Control 

A number of quality control (QC) procedures were implemented throughout the course of 
the BPS:04/ 09 student interview data collection. These procedures included frequent interview 
monitoring of telephone interviewers, a help desk that tracked and resolved difficulties encountered 
by sample members attempting to complete the web interview, quality circle feedback meetings, and 
FID A, interviewer, and tracer debriefings at the conclusion of the study. 

3.3.1 Interview Monitoring 

Regular monitoring of telephone interviews during BPS:04/ 09 data collection was 
conducted to meet the following important data quality objectives: 

• identification of problem items in the interview; 

• reduction in the number of interviewer errors; 

• improvement in interviewer performance through reinforcement of effective strategies; 
and 

• assessment of the quality of the data collected. 

Quality control supervisors (QCS) at Call Center Services and project staff monitored live 
and recorded interviews throughout data collection, using remote monitoring telephones and 
computer equipment. To guarantee an accurate reflection of data collection activities, QCS 
monitored day, evening, and weekend shift interviewers. In addition, each week QCS and interview 
project staff monitored one live interview session and one recorded interview session. The live 
session allowed for monitoring of calls and interviews in progress, including remotely viewing 
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interviewers’ computer screens as they progressed through the interview and listening to interviews 
in real time, while the session with recorded interviews allowed only listening to the interview but 
guaranteed an opportunity to hear complete interviews. QCS and interview project staff recorded 
observations on standardized monitoring forms that covered such topics as interviewer 
professionalism, question administration, and knowledge of the instrument. After each monitoring 
session, interviewers received feedback based on observations from the session. Issues and trends 
identified during monitoring were frequently incorporated into quality circle meetings to improve 
the quality of telephone interviews. 

3.3.2 Help Desk 

A help desk, described in section 3.2.1, was available to sample members. To gain a better 
understanding of the problems encountered by sample members, HDAs used a web-based 
application to record each help desk incident that occurred during data collection. For each incident, 
an HDA confirmed contact information for the sample member, noted the source (e.g., incoming 
telephone call, voice mail, or e-mail; request from the study website), recorded the type of problem, 
provided a description of the problem and resolution, and indicated the incident status (pending or 
resolved). If the problem was not resolved immediately, the HDA scheduled a follow-up 
appointment. 

Table 9 provides a summary of help desk incidents encountered during BPS:04/09 data 
collection. HDAs handled a total of 780 help desk incidents. The most common type of incident 
was from sample members requesting their study ID, password, or both (69 percent). Miscellaneous 
issues, including requests to complete the interview over the telephone, were the second most 
common category (16 percent). An additional 8 percent of incidents dealt with pop-up blocker 
issues. Three percent of the incidents reported were related to browser settings and computer 
problems. Incidents in which sample members indicated that the study website was down or 
unavailable represented 2 percent of all help desk requests, as did questions about the study. 
Incidents involving questionnaire content and instrument errors each accounted for less than 
1 percent of help desk incidents. 


Table 9. Help desk requests, by type of incident reported: 2009 


Type of incident reported 

Help desk requests 


Number 

Percent 

Total 

780 

100.0 

Study ID/password 

540 

68.6 

Pop-up blocker issues 

60 

8.2 

Browser settings/computer 

20 

2.6 

Website down/unavailable 

20 

2.3 

Questions about the study 

20 

1.9 

Questionnaire content 

# 

0.4 

Instrument error 

# 

0.3 

Other requests, not classifiable 

120 

15.8 


# Rounds to zero. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 
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3.3.3 Quality Circle Meetings 

Quality circle meetings were essential as part of a feedback loop for ensuring that project 
staff, CCS, and telephone interviewers were communicating on a regular basis about the goals of the 
study and addressing challenges encountered along the way. These meetings provided a fomm for 
discussing elements of the instmment design and interview cooperation tactics, motivating the group 
toward the goals of the study, and acquiring feedback on data collection issues. Weekly quality circle 
meetings for telephone staff were held at the call center, while quality circle meetings for the field 
staff were held via conference call. Issues discussed at these meetings were added to weekly notes, 
which all interviewers were required to access electronically. These quality circle notes included 
counts of interview completions to date, separate sections for general data collection issues and 
issues specific to the survey instmment, and project staff responses to questions from interviewers. 

Throughout the study, a variety of issues were addressed at the quality circle meetings that 
reinforced specific content from training and contributed to prompt problem solving. Some of the 
issues covered in these meetings included the following: 

• clarification of questions and item responses and reinforcement of positive interviewing 
techniques; 

• methods of gaining cooperation from sample members and gatekeepers (e.g., parents and 
roommates); 

• problem sheets submitted during interviews; 

• the importance of providing and reviewing detailed case comments; 

• data security protocols; and 

• study progress and general morale boosting. 

3.3.4 Debriefing 

At the conclusion of the BPS:04/09 data collection, project staff held debriefing meetings 
with interviewers, HDAs, and tracers to learn more about their experiences, and administered an 
anonymous online survey of the interviewers. With regard to tracing and locating strategies, 
interviewers believed that lack of land telephone lines among sample members was in fact a 
hindrance to tracing and locating but thought that contacting relatives was generally a helpful means 
of finding sample members. Also, their interactions with sample members led interviewers to 
emphasize that offering incentives seemed to positively affect participation in the survey. With 
regard to interview administration and content, debriefing feedback was typically positive, with 
interviewers identifying specifics of questions that presented challenges to some sample members 
when read over the phone. Project staff prepared summaries of the debriefing meetings and online 
survey for consideration when planning future administrations of BPS. 

3.4 Student Interview Data Collection Outcomes 

This section provides the results of the BPS:04/ 09 student interview data collection. Details 
surrounding the overall student interview response rate of 82 percent are included, and a description 
of the success of various locating methods is also provided. A timing analysis shows that the student 
interview, on average, took about 20 minutes to complete. 
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3.4.1 Student Interview Response Rates 

BPS:04/09 interviews were administered between February 24, 2009, and October 12, 2009. 
Of the 18,610 sample members remaining at the start of the BPS:04/09 student interview data 
collection, 16,920 (91 percent) were successfully located and asked to complete the BPS:04/09 
interview, while 15,160 (82 percent) did complete a full interview, an English or Spanish abbreviated 
interview, or a partial interview. 12 The overall locating and interviewing results for the BPS:04/09 
interview data collection effort, including sample members who were located but later considered 
exclusions for reasons such as being incapacitated or deceased, are presented in figure 8. 

Figure 8. Overall locating and interviewing results: 2009 


Sample 
n = 18,610 


1 

Not located 
n= 1,690 


_C 

Respondent 
r? = 15,160 


Nonrespondent n= 1,050 
Refusal n = 400 


Full interview 

n = 13,620 

English abbreviated 

n = 1,260 

Spanish abbreviated 

n = 210 

Partial 

n = 80 


Out of country 

n = 140 

Unavailable 

n = 50 

Deceased 

n = 70 

Institutionalized/incarcerated 

n = 40 

Incapacitated 

n = 20 


Located 

nonrespondent 

n = 1 .440 


Exclusions 
n = 320 


Located 
n = 16,920 


NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

Response rates by first-follow up (BPS:04/06) interview response status and 
institution type. First- follow up interview (BPS:04/ 06) respondents were located for and 
completed the second follow-up (BPS:04/ 09) interview at a higher rate than did first follow-up 
interview nonrespondents. 13 First follow-up interview respondents had a locate rate of 94 percent in 
the second follow-up interview, while the locate rate for first follow-up interview nonrespondents in 
the second follow-up interview was 80 percent. Of all first follow-up interview respondents, 

87 percent completed the second follow-up interview, while 61 percent of first follow-up interview 
nonrespondents completed the second follow-up interview. 


12 Partial interview status was assigned to any case where the respondent began a full interview and got through at least the enrollment 
history section but broke off and did not return to complete the interview. 

13 A completed interview is a full interview, an English or Spanish abbreviated interview, or a partial interview. 
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Overall locate rates of the BPS:04 cohort for the second follow-up interview, based on the 
institution type of the sample member’s base-year interview (NPSAS) school, ranged from 
81 percent at private for-profit less-than-2-year schools to 96 percent at private nonprofit 4-year 
doctorate-granting schools. Overall response rates for the BPS:04 cohort to the second follow-up 
interview, by NPSAS institution type, ranged from 70 percent at private for-profit less-than-2-year 
schools to 88 percent at public 4-year doctorate-granting schools and private nonprofit 4-year 
doctorate-granting schools. Locating and participation results, by first follow-up interview 
respondent status and NPSAS institution type, are presented in table 10. 

Table 10. Locating and participation rates, by first follow-up response status and NPSAS 
institution type: 2009 

Located Interviewed 


First follow-up response status and Percent Percent 

NPSAS institution type Total Number of total Number of total 


Total 

18,610 

16,920 

90.9 

15,160 

81.5 

First follow-up response status 

Respondent 

14,900 

13,940 

93.6 

12,890 

86.5 

Nonrespondent 

3,720 

2,980 

80.3 

2,280 

61.2 

NPSAS institution type 

Public 

Less-than-2-year 

600 

520 

86.0 

460 

75.7 

2-year 

6,380 

5,740 

90.1 

5,050 

79.1 

4-year non-doctorate-granting 

1,600 

1,490 

93.3 

1,360 

85.1 

4-year doctorate-granting 

3,080 

2,940 

95.4 

2,710 

87.8 

Private nonprofit 

2-year-or-less 

550 

470 

86.4 

410 

75.2 

4-year non-doctorate-granting 

2,210 

2,080 

94.4 

1,930 

87.2 

4-year doctorate-granting 

1,580 

1,520 

96.3 

1,400 

88.4 

Private for-profit 

Less-than-2-year 

1,570 

1,270 

80.9 

1,100 

70.4 

2 years or more 

1,050 

880 

84.1 

760 

72.2 


NOTE: Detail may not sum to totals because of rounding. NPSAS = National Postsecondary Student Aid Study. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


Response rates by first follow-up (BPS:04/06) interview response status and 
interview type. About 90 percent of all second follow-up (BPS:04/09) interview respondents 
completed the full second follow-up interview, about 1 0 percent completed an English or Spanish 
abbreviated second follow-up interview, and less than 1 percent completed a second follow-up 
partial interview. Ninety-one percent of first follow-up (BPS:04/ 06) interview respondents 
completed a full second follow-up interview, compared with 83 percent of first follow-up interview 
nonrespondents. Seven percent of first follow-up interview respondents completed a second follow- 
up English abbreviated interview, compared with 16 percent of first follow-up interview 
nonrespondents. Table 1 1 provides detail on the number and percent of completed second follow- 
up interviews, by first follow-up interview response status and by second follow-up interview type. 
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Table 11. Completed interviews, by first follow-up response status and interview type: 2009 

First follow-up response status 


Total Respondents Nonrespondents 


Interview type 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Total 

15,160 

100.0 

12,890 

100.0 

2,280 

100.0 

Full 

13,620 

89.8 

11,740 

91.1 

1,880 

82.5 

English abbreviated 

1,260 

8.3 

900 

7.0 

360 

15.7 

Spanish abbreviated 

210 

1.4 

190 

1.5 

20 

0.8 

Partial 

80 

0.5 

60 

0.4 

20 

0.1 


NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students Longitudinal Study 
(BPS:04/09). 


Interview outcomes by mode. BPS:04/ 09 interviews were completed in one of three 
modes: web, telephone, or field. Figure 9 shows that most (64 percent) interviews were completed 
on the web, 32 percent of interviews were completed by telephone, and 4 percent were completed in 
field interviewing. 

Figure 9. Percentage of completed interviews, by mode of administration: 2009 



NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


Response by phase of data collection. Half of all completed BPS:04/ 09 interviews were 
completed during the early response phase. Approximately 1 8 percent of all interviews were 
completed during the production phase, and the remaining 32 percent were completed during the 
nonresponse conversion phase. Response, by phase of data collection, is shown in figure 10. 
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Figure 10. Percentage of completed interviews, by data collection phase: 2009 



NOTE: Partial interviews were not included because partially completed interviews could be resumed by sample members through 
the end of data collection. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

The early response phase of data collection yielded a 40 percent response rate, with 7,470 
completed interviews out of 18,610 cases (table 12). During this phase of data collection, first 
follow-up (BPS:04/ 06) interview respondents received $30 to complete the second follow-up 
(BPS:04/ 09) student interview, and first follow-up interview nonrespondents received $50 to 
complete the second follow-up interview. The next phase of data collection, the production phase , 
yielded a 25 percent response rate, with 2,760 completed interviews out of the remaining 11,140 
cases. During the production phase, first follow-up interview respondents received $20 to complete 
the second follow-up interview, and first follow-up interview nonrespondents received $40 to 
complete the second follow-up interview. The final phase of data collection, the nonresponse conversion 
phase, yielded a 58 percent response rate, with 4,860 completed interviews out of the final remaining 
8,380 cases. Respondents who completed the second follow-up interview during this last phase of 
data collection received either $30 if they were first follow-up interview respondents or $50 if they 
were not. 


Table 12. Number of cases and percentage of completed interviews within each phase of data 
collection: 2009 


Data collection phase 

Number of 

cases 

Completed interviews 

Number Percent of cases 

Total 

18,610 

15,080 

81.0 

Early response phase ($30/$50) 

18,610 

7,470 

40.1 

Production phase ($20/$40) 

11,140 

2,760 

24.8 

Nonresponse conversion phase ($30/$50) 

8,380 

4,860 

57.9 


NOTE: Detail may not sum to totals because of rounding. Partial interviews are not included because partial status 
could not be assigned until the end of data collection. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 

Locate rates by source of address update. Address updates for the BPS:04 cohort were 
received from 7,910 sample members in response to the panel maintenance mailing, the parent 
mailing (for sample members younger than 26 years old), the advance notification mailing, or 
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through the BPS website. If an address update was received, the sample member was located almost 
100 percent of the time and completed interviews 98 percent of the time. The parent mailing elicited 
the most address updates, with the other two mailings producing about equal numbers of address 
updates. Locating outcomes of cases for which an address update was provided are shown in table 
13. 


Table 13. Located and interview completion rates, by source of address update: 2009 


Source of address update 

Number that 
provided an update 

Located 

Number 

Percent 

Interviewed 

Number Percent 

Total 

7,910 

7,850 

99.3 

7,740 

97.8 

Panel maintenance mailing 

1,900 

1,880 

99.4 

1,870 

98.4 

Parent mailing 

2,990 

2,960 

99.0 

2,900 

97.0 

Advance notification mailing 

1,970 

1,960 

99.2 

1,930 

97.9 

B&B Student website reply 

1,060 

1,050 

99.9 

1,040 

98.8 


NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


Response rates by intensive tracing method. Among the cases assigned to intensive 
tracing, TOPS-1 and TOPS-2, approximately 63 percent were located. Of those cases located 
through intensive tracing, about 56 percent completed the BPS:04/09 interview (table 14). 

Table 14. Located and interviewed rates of cases requiring intensive tracing, by intensive tracing 
method: 2009 


Intensive tracing method 


Located 

Interviewed 

Total 

Number 

Percent of 
total cases 

Number 

Percent of 
located cases 

Total 

3,180 

2,010 

63.2 

1,130 

56.3 

TOPS-1 

3,180 

1,910 

60.1 

1,080 

56.7 

TOPS-2 1 

440 

210 

47.4 

110 

51.4 


NOTE: Detail may not sum to totals because of rounding. TOPS = tracing operations. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


Locate rates by other locating methods. Although no significant locating outcomes were 
achieved through the use of social networking sites, 60 percent of the 7,060 cases traced with 
Experian and FastData batch searches were located. Experian MetroNet batch searches confirmed 
or provided new contact information for 3,620 cases. More than one-third of the cases sent through 
the Experian credit header searches and the FastData Superphone search were located. Results of 
these other locating methods are summarized in table 15. 
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Table 15. Locate rates, by other locating methods: 2009 


Tracing source 

Cases traced 

Located 

Number 

Percent of 
cases traced 

Total 

7,060 

4,200 

59.5 

Experian MetroNet batch searches 

5,420 

3,620 

66.8 

Experian credit header searches 

1,020 

370 

36.5 

FastData Superphone search 

620 

210 

33.4 


NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


Of the 130 sample members who received an early text message notifying them that data 
collection had begun, 95 percent completed the interview (figure 1 1). These were sample members 
who had requested a text notification reminder of the start of data collection on their address 
updates. Near the end of data collection, text messaging was again employed as an additional means 
of contact for sample members who had become difficult to reach by other contacting methods and 
for whom there was a cell phone number on record. Of the 1,530 cases sent a text message near the 
end of data collection, 22 percent completed the interview. 


Figure 11. Interview completion rates, by timing of text messages: 2009 

Percent 



Early in the study End of study 

Text messages sent 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

Prompting response rates. First follow-up (BPS:04/ 06) interview nonrespondents were 
selected to receive prompting calls to complete the web interview during the early response phase of 
the second follow-up (BPS:04/09) data collection. Of the 3,720 first follow-up interview 
nonrespondent cases, 72 percent of the cases, or 2,680 cases, were flagged for prompting. The 
remaining 1,040 first follow-up interview nonrespondent cases were not flagged for prompting 
because of either invalid telephone numbers or because a second follow-up interview was completed 
prior to the start of prompting. Prompting calls began 10 days after the start of data collection, and 
1,540 of the cases were prompted. The response rate during the early response phase of data 
collection was significantly higher for sample members spoken to directly in a prompting call 
(21 percent), compared with sample members who received a prompting call but who were not 
spoken to directly (6 percent; % = 1 ,99,p < .001). A sample member was considered not contacted 
when someone other than the sample member was spoken to or when the call was directed to an 
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answering machine or voice mail. The response rates during the early response phase of data 
collection for first follow-up interview nonrespondents who were reached directly when prompted 
versus those who were not contacted are compared in figure 12. 

Figure 12. Early response rates for first follow-up interview nonrespondents who were prompted: 
2009 


Early response rate 



Prompted first follow-up nonrespondents 

1 A sample member was considered not contacted when someone other than the sample member was reached or the call was 
directed to an answering machine or voice mail. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

Response rates for field cases. Beginning about 4 months into data collection, 1,430 cases 
identified as difficult to locate but whose last known address was in a selected field cluster were 
assigned to field data collection. Sixty-three percent of these cases were located, and 54 percent 
completed the interview in one of the three available modes (web, telephone, or field). Of the 
completed cases originally assigned to the field, 170 (22 percent) were actually completed by web, or 
by telephone through the call center, instead of through field efforts. 

Results by first follow-up (BPS:04/ 06) interview response status and NPSAS institution type 
are also presented for these cases. Nearly 63 percent of the first follow-up interview respondent 
cases assigned to the field completed the second follow-up (BPS:04/09) interview, compared with 
47 percent of the first follow-up interview nonrespondent cases assigned to the field who completed 
the second follow-up interview. 

Response rates among cases assigned to the field also varied by sample members’ NPSAS 
institution type. The largest number of cases assigned to the field for any one institution type, 580 
cases, was for sample members whose NPSAS institution was a public 2-year institution; these cases 
yielded a response rate of 57 percent. Only 60 cases were assigned to the field for sample members 
from public less-than-2-year schools; these cases yielded a response rate of 43 percent. Table 16 
shows locate and response rates among the field cases. 
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Table 16. Located and completed field interview cases, by first follow-up response status and institution type: 2009 



Number 

Located 


Number completed interviews 

Percent completed interviews 

First follow-up response status and 
institution type 

assigned to 
field 

Number 

Percent 

Field 

Web or 
help desk 

All 

modes 

Field 

Web or 
help desk 

All 

modes 

Total 

1,430 

900 

63.0 

610 

170 

780 

42.8 

11.5 

54.4 

First follow-up response status 

Respondent 

650 

460 

70.3 

300 

110 

410 

46.6 

16.2 

62.8 

Nonrespondent 

780 

440 

56.8 

310 

60 

370 

39.7 

7.6 

47.3 

Institution type 

Public 

Less-than-2-year 

60 

30 

47.6 

20 

10 

30 

31.7 

11.1 

42.9 

2-year 

580 

380 

65.5 

270 

60 

330 

45.9 

10.9 

56.8 

4-year non-doctorate-granting 

80 

50 

61.7 

30 

10 

40 

35.8 

16.0 

51.9 

4-year doctorate-granting 

80 

60 

73.3 

30 

10 

40 

40.0 

17.3 

57.3 

Private nonprofit 

2-year-or-less 

60 

30 

55.9 

20 

10 

30 

40.7 

10.2 

50.8 

4-year non-doctorate-granting 

90 

60 

68.2 

30 

20 

40 

31.8 

18.8 

50.6 

4-year doctorate-granting 

60 

40 

75.0 

20 

10 

30 

35.7 

16.1 

51.8 

Private for-profit 

Less-than-2-year 

270 

160 

57.0 

130 

20 

150 

46.0 

7.7 

53.7 

2 years or more 

160 

100 

61.5 

70 

20 

90 

44.7 

10.6 

55.3 


NOTE: Detail may not sum to totals because of rounding. Sample members were still able to access the web instrument and call the help desk to complete a telephone interview once 
a case was sent to the field. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). 
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3.4.2 Interview Timing Burden 

Several analyses for the BPS:04/09 interview assessed the timing burden on respondents 
completing the interview. These analyses included computation of the overall average time it took 
respondents to complete the interview and the time it took respondents to complete the interview 
based on mode of administration (web, telephone, or field) and analysis of whether particular 
respondent characteristics, such as employment status, were related to interview timing burden. 

To calculate the time it took to complete the interview, two time stamps were embedded on 
each form (web screen) of the interview. A start timer recorded the clock time on the respondent’s 
or interviewer’s computer when each form was first displayed. An end timer recorded the clock time 
on the respondent’s or the interviewer’s computer when the Next button on each form was clicked. 
From the two time stamp variables, an on-screen time and a transit time were calculated. The 
on-screen time was calculated by subtracting the start time from the end time for each form that the 
respondent saw. The transit time was calculated by subtracting the end time of the preceding form 
from the start time of the next form. Total on-screen time and total transit time were calculated for 
all respondents by summing all of the on-screen times for each screen received and summing all of 
the transit times for each respondent. Total instmment time was then calculated by summing a 
respondent’s total on-screen and total transit times. 

The timing analysis included only cases that completed the full-scale interview in one 
session. Partially completed interviews and those interviews completed in multiple sessions (i.e., 
those cases that logged out from an incomplete interview and later resumed their interviews) were 
excluded from the analysis. 

The average overall interview time was calculated by summing the respondents’ interview 
completion times and dividing the result by the total number of respondents. On average, the 
BPS:04/09 interview took 19.6 minutes to complete. Web interviews took 19.4 minutes to complete, 
field interviews took 19.6 minutes to complete, and telephone interviews took 20.2 minutes to 
complete. 

Average section completion times were 1.3 minutes for the Front End (introductory 
section), 4.1 minutes for Enrollment Flistory, 4.5 minutes for Enrollment Characteristics, 

6.1 minutes for Employment, and 4.1 minutes for Background. Notably, between web and 
telephone modes, the Enrollment History section took longer in web mode than telephone mode 
(/(6,700) = 2.19, p < .01), as did the Enrollment Characteristics section (/(8,060) = 7.79,p <.001), 
but the Background section took longer in telephone mode than in web mode (/(l 1,480) = 22.06, 
p < .001). Table 17 shows the average interview time overall, for each section, by mode of 
administration. 
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Table 17. Average time in minutes to complete interview section, by mode of administration: 2009 


Interview section 

All respondents 

Mode of administration 

Web 

Telephone 

Field 

Number 
of cases 

Average 

time 

Number 
of cases 

Average 

time 

Number 
of cases 

Average 

time 

Number 
of cases 

Average 

time 

Total interview 

11,780 

19.64 

7,780 

19.39 

3,700 

20.15 

290 

19.65 

Front End 

11,480 

1.27 

7,780 

0.94 

3,700 

1.98 

t 

T 

Enrollment History 

11,780 

4.08 

7,780 

4.15 

3,700 

3.97 

290 

3.38 

Enrollment Characteristics 

11,780 

4.45 

7,780 

4.61 

3,700 

4.17 

290 

3.96 

Employment 

8,600 

6.06 

5,340 

6.08 

2,970 

6.09 

290 

5.51 

Background 

11,780 

4.13 

7,780 

3.88 

3,700 

4.60 

290 

4.96 


t Not applicable. 

NOTE: The timing analysis included only cases that completed the interview in one session; partial interviews were excluded. Detail 
may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

Interview time by interview path. The time it took respondents to complete the 
BPS:04/ 09 interview varied by the number of schools attended as of the last follow-up in 2006 and 
by whether respondents were employed because more questions were asked of those respondents 
who attended more than one school and who had jobs. 

The Enrollment History section collected information about the respondent’s enrollment 
and degree attainment since the last follow-up in 2006 (for more about the interview design and 
topics covered in each interview section, see section 3.1.1). The average time it took all respondents 
to complete this section was 4.1 minutes. Because the interview collected information on each 
school attended since 2006, respondents with more schools spent more time in this section. 

The Enrollment Characteristics section collected information about the respondent’s 
experiences since 2006 while enrolled as an undergraduate. Topics focused on major or field of 
study, grade point average, employment while enrolled, and financial aid. This section took an 
average of 4.5 minutes to complete. Respondents who had no additional schools since 2006 took an 
average of only 1.8 minutes in this section because they had no additional undergraduate enrollment 
since the base-year study on which to report. Table 18 shows the average interview time to complete 
the enrollment sections by the number of schools attended. 

Table 18. Average time in minutes to complete enrollment sections, by number of schools 
attended: 2009 


Number of schools attended Number Average completion time 

since 2006 of cases Enrollment History Enrollment Characteristics 


None 

3,480 

0.87 

1.84 

One 

7,210 

4.98 

5.57 

Two 

1,090 

8.37 

5.45 


NOTE: The timing analysis included only cases that completed the interview in one session; partial interviews were 
excluded. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 

Respondents took an average of 6 minutes in the employment section to report on job items 
such as their job title and duties, benefits, earnings, level of satisfaction, and any periods of 
unemployment. Respondents who were employed spent 7.4 minutes in the employment section, 
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compared with those respondents who were not employed, who spent 1.6 minutes in the 
employment section. 

Timing of abbreviated interview. The abbreviated version of the BPS:04/ 09 interview 
included the Front End, Enrollment History, and Enrollment Characteristics sections of the 
interview. On average, the BPS:04/09 abbreviated interview took 10.5 minutes. Overall, web 
abbreviated interviews took 11.3 minutes and were significantly longer than telephone abbreviated 
interviews, at 10.5 minutes (/(1,120) = 2.0 ,p < .05), and field abbreviated interviews, at 8.9 minutes 
(/(620) = 4.2 ,p < .001). Also, telephone abbreviated interviews were significantly longer than field 
abbreviated interviews (/(830) = 3.1 , p < .01). Table 19 shows the average abbreviated interview time 
overall, for each section, and by interview completion mode. 


Table 19. Average time in minutes to complete abbreviated interview, by interview section and 
mode of administration: 2009 


Interview section 

All respondents 

Mode of administration 

Web 

Telephone 

Field 

Number 
of cases 

Average 

time 

Number 
of cases 

Average 

time 

Number 
of cases 

Average 

time 

Number 
of cases 

Average 

time 

Total interview 

1,290 

10.54 

460 

11.26 

670 

10.49 

170 

8.85 

Front End 

1,120 

1.9 

460 

1.17 

670 

2.4 

t 

t 

Enrollment History 

1,290 

3.5 

460 

3.61 

670 

3.53 

170 

3.06 

Enrollment 









Characteristics 

1,290 

4.29 

460 

4.69 

670 

4.15 

170 

3.74 

Employment 

t 

t 

t 

t 

t 

t 

t 

t 

Background 

t 

t 

t 

t 

t 

t 

t 

t 


t Not applicable. 

NOTE: The timing analysis included only cases that completed the interview in one session; partial interviews were excluded. Detail 
may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

3.4.3 Telephone Interviewer Hours 

During the course of BPS:04/09 data collection, 15,195 telephone interviewer hours were 
logged, for an average of 3.09 hours spent by telephone interview staff per completed telephone 
interview. Because, on average, telephone interviews took 20.2 minutes to administer, most of the 
telephone interview hours were spent on case management activities such as locating and contacting 
sample members, prompting sample members to complete interviews, reviewing call history, 
scheduling callbacks, entering detailed comments and suggestions to assist with reaching and 
interviewing sample members, and responding to incoming help desk calls. 

3.4.4 Number of Calls to Sample Members 

On average, 16 calls were made per BPS:04/09 sample member during the interviewing 
period, except for in the early response phase when no outbound calls were made to sample 
members. The average number of calls per sample member varied according to the sample 
member’s first follow-up (BPS:04/ 06) interview response status and second follow-up (BPS:04/ 09) 
interview response status, mode of administration, and the phase of data collection. First follow-up 
interview respondents received 7 fewer calls, 1 5 calls on average, than first follow-up interview 
nonrespondents who received 22 calls on average (7(5,305) = 15.01, < .001). Cases that completed 
a second follow-up interview received an average of 1 1 calls, while second follow-up interview 
nonrespondents received an average of 42 calls during the interviewing period. 
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There were also call count differences depending on mode of interview administration. 
Overall, respondents who completed interviews over the telephone required more calls than 
respondents who completed interviews over the web, an average of 1 8 calls, compared with 7 calls 
(/(8,240) = 35.46, < .001). However, when early response phase interview completes were 

excluded, web respondents required more calls than telephone respondents, an average of 24 calls, 
compared with 18 calls (7(5,144) = 13.23,^> < .001). The average number of telephone calls is shown 
in table 20. 


Table 20. Average number of calls, by response status and data collection phase: 2009 


Response status and data collection phase 

Number of 

cases 

Number of 
calls 

Average 
number of calls 

Total 

18,610 

302,500 

16.3 

First follow-up interview response status 

Respondent 

14,900 

221,340 

14.9 

Nonrespondent 

3,720 

81,150 

21.8 

Second follow-up interview response status 

Respondent 

15,160 

158,730 

10.5 

Web interviews 

9,630 

64,060 

6.7 

Excluding early response 

2,670 

63,960 

23.9 

Telephone interviews 

4,920 

86,760 

17.6 

Field interviews 

610 

7,910 

12.9 

Nonrespondent and exclusions 

3,450 

143,770 

41.7 


NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


3.5 Evaluation of Student Interview Items 

An evaluation of the BPS:04/ 09 student interview items included analyses of the data 
collected in the instrument coders and a review of help text access rates, success rates for conversion 
text, and item nonresponse. 

3.5.1 Instrument Coders 

Assisted coding systems were used to standardize the collection of data on, and to code, any 
postsecondary schools attended, major or field of study, and occupation. Text strings were collected 
from the respondent, and then a keyword search of an underlying database was conducted, allowing 
the respondent to select the best option from a list of possible options returned. An assisted coding 
system was not used to code industries, but ready industry definitions allowed respondents and 
interviewers to select an industry classification from among a list of standardized options (for a 
detailed description of each coder, see section 3.1.1). 

Recoding. Ten percent of the major, occupation, and industry codes chosen in the student 
interviews were randomly selected to be reviewed by expert coding staff for recoding . 14 Expert coders 
assessed the accuracy of codes chosen in the interview based on the text string provided by the 
respondent to review data and to provide information for improving instrument coders for future 


14 Recoding of institution codes selected in the postsecondary school coders was not done because text strings provided by 
respondents would presumably have directly matched school name codes chosen. In the major, occupation, and industry coders, text 
strings provided by respondents and standardized names of codes in the database were often not direct matches. 
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studies. Across modes of administration and across coders, expert coding staff generally agreed with 
the codes chosen for text strings in the interview. Overall, expert coders agreed with major, 
occupation, and industry codes chosen in the interview 92 percent of the time, recoded codes 
chosen to a new value about 7 percent of the time, and were unable to choose a code based on too 
vague a text string about 1 percent of the time. 

Only the industry coder showed significant differences in recode rates between modes of 
administration. Expert coders agreed with industry codes chosen by web respondents 72 percent of 
the time and with those chosen by interviewers 92 percent of the time = 7.28, _/> < .001). Expert 
coders recoded industry codes chosen by web respondents 23 percent of the time and recoded those 
chosen by telephone and field interviewers 7 percent of the time = 6.18,^ < .001). Industry text 
strings provided by web respondents were too vague to code 5 percent of the time, while industry 
text strings from web and field interviews were too vague to code 1 percent of the time (% = 3.51, 
p < .001). Table 21 shows the rate of recoded values — same as original code, recoded to different 
value, or text string too vague to code — chosen by the expert coders for the major, occupation, and 
industry codes in the interview. 


Table 21. Summary of recoding results, by coder type and administration mode: 2009 


Coding system 



Percent of recoded values 



Recoded same as 
original 

Recoded to different 
value 

Text string too vague 
to code 

Web 

Telephone 
and field 

Web 

Telephone 
and field 

Web 

Telephone 
and field 

Major 

97.9 

98.0 

1.6 

2.0 

0.5 

# 

Occupation 

90.2 

92.9 

9.0 

6.9 

0.8 

0.2 

Industry 

72.3 

91.7 

22.7 

7.4 

4.9 

0.9 


# Rounds to zero. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


Upcoding. In a process known as upending, project staff chose an appropriate code for any 
text strings provided by respondents or interviewers for which a code was not selected in the 
IPEDS, major, occupation, and industry instrument coders. Text strings from web interviews 
generally required more upcoding than text strings from telephone and field interviews because 
interviewers received special training on coders. Results of the upcoding process are shown in table 
22 . 

Table 22. Summary of upcoding results, by coding system and administration mode: 2009 


Percent of text strings upcoded 


Coding system 

Overall 

Web 

Telephone and field 

IPEDS institutions 

3.8 

3.9 

3.7 

Major 

2.6 

3.4 

0.6 

Occupation 

3.7 

5.2 

0.9 

Industry 

1.9 

2.9 

0.1 


NOTE: IPEDS = Integrated Postsecondary Education Data System. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 
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3.5.2 Help Text 

Respondents or interviewers were able to click on a help button provided on each 
BPS:04/ 09 interview screen for both general instrument and question-specific help. The general 
instrument help provided answers to FAQs about web browser settings and response types (i.e., 
how to respond using a check box, dropdown box, or radio button). The question-specific help 
provided definitions of key terms and phrases used in question wording and response options and 
also provided any other explanations thought to help clarify and standardize the meaning of 
questions for respondents. 

The number of times that respondents or interviewers clicked the help button on each 
screen relative to the number of respondents who were administered the question determined the 
rate of help text access for that screen. The screen-level rate of help text access was analyzed overall 
and by mode of interview administration to identify screens that may have been problematic for 
users. For forms administered to at least 50 respondents, the overall mean rate of help text hits per 
screen was less than 1 percent. Flelp text was accessed 2 percent of the time during interviews by 
telephone and field, compared with 1 percent of the time by web respondents = \2A9,p < .001). 

The interview question asking respondents for their employer’s primary industry (item name 
MDINDCD) had the highest overall rate of help text access, at 7 percent. The help text for this 
question was accessed 19 percent of the time by telephone and field interviewers, compared with 
less than 1 percent of the time by web respondents = 29.9 6,p < .001). It is worth noting here that 
interviewers were encouraged as part of their training to refer to the help text on this item when 
respondents showed hesitation in understanding the term “industry.” 

Table 23 shows the interview questions with the highest rates of help text access; these were 
questions administered to at least 50 respondents and for which help text was accessed at a rate of at 
least 2 percent. 

Table 23. Interview questions with highest rates of help text access, by administration mode: 2009 

Telephone and 

Overall Web field 

Percent Percent Percent 


Question 

name 

Description of question 

Number 
adminis- 
tered to 

of help 
text 

access 

Number 
adminis- 
tered to 

of help 
text 

access 

Number 
adminis- 
tered to 

of help 
text 

access 

MCCLTYA 

Type of class 

Loan influenced enrollment 

800 

3.1 

450 

0.2 

350 

6.9 

MCLNINL 

plans 

4,630 

2.2 

2,610 

0.0 

2,030 

5.1 

MCLNTYA 

Type of undergraduate loan 
Undergraduate level during 

8,080 

3.0 

5,390 

0.0 

2,690 

8.9 

MCUGYR 

most recent term 

1,480 

2.4 

750 

0.3 

730 

4.5 

MDCURTYP 

Type of employer 

7,670 

3.2 

5,040 

0.0 

2,630 

9.4 

MDINDCD 

Industry coder 

6,700 

6.8 

4,370 

0.1 

2,340 

19.4 

MDNTCAA 

Job description 

2,480 

3.0 

1,540 

0.0 

940 

7.9 


NOTE: Detail may not sum to totals because of rounding. Table is based on the rates of help text access for interviewer screens 
administered to a minimum of 50 respondents and in which help text was accessed at an overall rate of at least 2 percent. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

3.5.3 Conversion Text 

Eight questions in the interview were considered critical ; that is, responses to these questions 
were especially important to the study and high rates of missing data on these questions would 
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impact the quality of these data. When respondents did not provide an answer to these questions 
and the Next button was clicked on the interview screen, then conversion language (or conversion text) 
appeared above the question to encourage a response. Interviewers were asked to read conversion 
text to respondents and then to reread the interview question. The conversion text attempted to 
relay the importance of that particular question to the study and emphasized the confidential nature 
of responses. Some critical questions also displayed a don ’t know response option for respondents 
once the conversion text was triggered. 

Dividing the total number of responses to the critical questions after the conversion text was 
displayed by the total number of cases where the conversion text was triggered provided a 
conversion rate for the questions that was attributed to the conversion text. Overall, conversion text 
was triggered in the student interview approximately 930 times throughout data collection. Seventy- 
six percent, or 700, of these cases were converted to a response after the conversion text was 
displayed. The web interviews accounted for 600 of the 930 cases where conversion text was 
triggered and 510 of the 700 converted cases. The remaining cases where conversion case was 
triggered were in telephone and field interviews. The rate of conversion as measured by the 
triggering of conversion text was 85 percent in web interviews, compared with 59 percent in 
telephone and field interviews = 8.67, _/> < .001). There was not a way to measure conversion to a 

response by telephone and field interviewers before conversion text was triggered. 

Conversion text was triggered more than 100 times for three interview questions. Enrollment 
through June 2009 (MBDAT01), which asked sample members to provide their months of 
postsecondary enrollment for the period from July 2006 through June 2009, triggered conversion 
text in 330 cases and yielded a conversion rate of 72 percent. Web cases were converted at a rate of 
86 percent, compared with telephone and field cases, which were converted at a rate of 50 percent 
= 7.24, p < .001). Monthly rent or mortgage payment amount (MEMTGAMT) triggered conversion text 
in 320 cases, with a conversion rate of 75 percent. Web cases were converted at a rate of 88 percent, 
compared with telephone and field cases, which were converted at a rate of 44 percent = 8.23, 
p < .001). Any undergraduate loans (MCUGLN) triggered conversion text in 130 cases and produced a 
conversion rate of 85 percent, with no significant mode difference. Table 24 displays the rates of 
conversion for all eight items in the interview with conversion text. Conversion rates were examined 
overall and by mode. 
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Table 24. Conversion rates for critical items, by mode of administration: 2009 







Percent 

Percent 





Total 

converted 

converted 



Number 

Number 

percent 

to a valid 

to a “don’t 

Item 

Item description 

of cases 

converted 

converted 

response 

know” 





Total 



MBDAT01 

Enrollment through June 2009 

330 

240 

72.4 

72.4 

t 

MCGAPEST 

Estimate of GPA 

Number of jobs during last term of 

60 

50 

76.3 

67.8 

8.5 

MCNUMJOB 

enrollment 

20 

20 

94.7 

94.7 

t 

MCUGLN 

Any undergraduate loans 

130 

110 

84.9 

80.2 

4.8 

MDJSTAT 

Currently employed 

20 

20 

90.0 

90.0 

t 

MECITZN 

Citizenship status 

10 

10 

71.4 

71.4 

t 

MECRDUSE 

Have credit cards 

Monthly rent or mortgage payment 

50 

30 

69.6 

69.6 

t 

MEMTGAMT 

amount 

320 

240 

74.9 

59.6 

15.4 





Web 



MBDAT01 

Enrollment through June 2009 

210 

180 

86.3 

86.3 

t 

MCGAPEST 

Estimate of GPA 

Number of jobs during last term of 

40 

30 

78.0 

73.2 

4.9 

MCNUMJOB 

enrollment 

10 

10 

92.9 

92.9 

t 

MCUGLN 

Any undergraduate loans 

70 

60 

81.1 

78.4 

2.7 

MDJSTAT 

Currently employed 

10 

10 

85.7 

85.7 

t 

MECITZN 

Citizenship status 

# 

# 

66.7 

66.7 

t 

MECRDUSE 

Have credit cards 

Monthly rent or mortgage payment 

30 

20 

66.7 

66.7 

t 

MEMTGAMT 

amount 

220 

200 

87.9 

70.1 

17.9 




Telephone and field 


MBDAT01 

Enrollment through June 2009 

130 

60 

49.6 

49.6 

t 

MCGAPEST 

Estimate of GPA 

Number of jobs during last term of 

20 

10 

72.2 

55.6 

16.7 

MCNUMJOB 

enrollment 

10 

10 

100.0 

100.0 

t 

MCUGLN 

Any undergraduate loans 

50 

50 

90.4 

82.7 

7.7 

MDJSTAT 

Currently employed 

10 

10 

100.0 

100.0 

t 

MECITZN 

Citizenship status 

# 

# 

75.0 

75.0 

t 

MECRDUSE 

Have credit cards 

Monthly rent or mortgage payment 

20 

10 

75.0 

75.0 

t 

MEMTGAMT 

amount 

100 

40 

44.2 

34.7 

9.5 


t Not applicable. 

# Rounds to zero. 

NOTE: Detail may not sum to totals because of rounding. GPA = grade point average. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

3.5.4 Item Nonresponse 

Rate of nonresponse was a data quality measure used to identify troublesome interview items 
and better understand the experiences of sample members in completing the interview. Total 
nonresponse rates were calculated for items with missing data (including don’t know responses) that 
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were administered to at least 100 respondents. Overall, the item-level nonresponse analysis yielded 
19 items, out of 385 interview items, with more than 5 percent missing data. 15 

Income questions returned relatively high rates of nonresponse. 16 The item with the highest 
rate of nonresponse was spouse’s income estimate for 2008 (MEINSRA). Of the 250 respondents who 
received this item, approximately 43 percent did not provide an estimate of their spouse’s income. In 
addition, there were several postbaccalaureate degree questions and items on coders among those 
items with nonresponse rates greater than 5 percent. Fifteen percent of respondents did not enter a 
text string for their job duties in the occupation coder item (MDJBDY), and 21 percent did not 
enter a text string for their employer’s industry in the industry coder item (MDIND). 

Item-level nonresponse rates were also examined by mode of administration. There were 
significant differences between the web mode and interviewer (telephone and field) modes in the 
nonresponse rates of 15 interview items. Notably, the following income and debt items showed 
higher rates of nonresponse among telephone and field respondents than among web respondents: 
respondent income in 2008 (MEINCOM; ^ = 0A2,p < .001); spouse’s income in 2008 (MEINCSP; 

^ = 5.26 ,p < .001); spouse’s income estimate for 2008 (MEINSRA; ^ = 2.12 ,p < .05); balance due on all 
credit cards (MECRDBAL; ^ = 3.41, p < .001); spouse’s total student loan amount (MESPAMT; ^ = 9.28, 
p < .001); and amount of spouse’s loans still owed (MESPOWE; ^ = 6.90 ,p < .001). 

In contrast, the following items on coders showed higher rates of nonresponse among web 
respondents than among telephone and field respondents: primary major: string (MCMAJ1; ^ = 17.84, 
p < .001); secondary major: string (MCMAJ2; ^ = 0.20, p < AQljjob duties (MDJBDY; ^ = 20.54, 
p < .001); industry: string (MDIND; ^ = 16.27 ,p < .001). Table 25 summarizes the item-level 
nonresponse for items administered to at least a 100 respondents with a rate of at least 5 percent 
missing. 


15 Partial and abbreviated interview completions and missing data for interview nonrespondents were excluded from this analysis. 

16 Results of item nonresponse in this section apply only to BPS:04/09 respondents and are not weighted by the BPS analysis weight. 
Chapter 6 provides additional details on the item nonresponse, the potential bias due to item nonresponse, and the impact of the item 
imputation which was used to fill in missing data to reduce item nonresponse bias. 
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Table 25. Interview items with highest nonresponse rates, by mode of administration: 2009 

Mode of administration 


Telephone and 

Overall Web field 


Item 

Item description 

Number 
adminis- 
tered to 

Percent 

missing 

Number 
adminis- 
tered to 

Percent 

missing 

Number 
adminis- 
tered to 

Percent 

missing 

Enrollment History 

MBEN803 Intensity of enrollment 2008 at school 3 

140 

6.6 

110 

6.3 

30 

7.7 

MBCRGR02 

Currently enrolled in postbaccalaureate 
program 2 

130 

7.6 

110 

6.1 

20 

16.7 

MBDGGR02 

Type of postbaccalaureate degree or 
certificate program 2 

170 

7.3 

150 

5.5 

20 

20.0 

MBTIGR02 

Enrollment intensity in postbaccalaureate 
program 2 

170 

6.1 

150 

4.1 

20 

20.0 

MBMRGR02 

Enrolled for any other postbaccalaureate 
degrees or certificates 2 

170 

6.1 

150 

4.8 

20 

15.0 

Enrollment Characteristics 

MCMAJ1 Primary major: string 

9,750 

8.6 

7,190 

11.6 

2,560 

0.1 

MCMAJ2 

Secondary major: string 

1,620 

6.4 

1,180 

8.7 

450 

0.2 

Employment 

MDJBDY 

Job duties 

7,740 

14.7 

5,090 

20.6 

2,650 

3.2 

MDIND 

Industry: string 

6,680 

21.3 

4,350 

27.3 

2,330 

10.2 

MDJOBYR 

Years in same or similar job 

7,740 

6.6 

5,090 

7.7 

2,650 

4.4 

MDJOBMM 

Months in same or similar job 

7,740 

14.4 

5,090 

15.5 

2,650 

12.1 

Background 

MEINCOM 

Respondent income in 2008 

13,600 

8.2 

9,090 

7.1 

4,510 

10.3 

MEINEST 

Respondent income estimate for 2008 

1,110 

17.7 

650 

20.1 

470 

14.4 

MEINCSP 

Spouse's income in 2008 

3,060 

8.1 

1,830 

6.0 

1,230 

11.3 

MEINSRA 

Spouse's income estimate for 2008 

250 

43.0 

110 

35.5 

140 

48.9 

MECRDBAL 

Balance due on all credit cards 

4,410 

5.0 

3,190 

4.3 

1,220 

6.8 

MESPAMT 

Spouse’s total student loan amount 

1,070 

9.0 

750 

3.6 

330 

21.2 

MESPOWE 

Amount of spouse’s loans still owed 

1,070 

6.2 

750 

2.8 

330 

13.8 


NOTE: Detail may not sum to totals because of rounding. This table includes only those items that were administered to at least 100 
respondents. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


3.6 Student Interview Conclusions 

BPS:04/09 interviews were conducted from February 24, 2009, to October 12, 2009. Of the 
18,610 sample members in the BPS:04 cohort, 16,920, or 91 percent, were successfully located. 
Successful locating methods included batch searches, such as Telematch and CPS, and address 
update information provided by both sample members and their parents. Overall, about 17 percent 
of sample members required intensive tracing, and 63 percent of these cases were located. New 
locating methods attempted during the BPS:04/ 09 interview included text message reminders and 
the use of social networking sites. While the use of social networking sites was abandoned as a 
means to contact sample members, approximately 22 percent of those sample members who 
received a reminder text message near the end of data collection responded to the interview. 

Of the 18,610 sample members in the BPS:04 cohort, 15,160, or 82 percent, completed a 
full, abbreviated (English or Spanish), or partial interview. About 90 percent of all second follow-up 
(BPS:04/ 09) interview respondents completed the full interview. Sixty- four percent of interviews 
were completed on the Web, 32 percent of interviews were completed by telephone, and 4 percent 
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were completed in field interviewing. Eighty-seven percent of first follow-up (BPS:04/ 06) interview 
respondents completed the second follow-up interview, compared with 6 1 percent of first follow-up 
interview nonrespondents. Half of all completed second follow-up interviews and almost three- 
fourths of web interviews were completed during the early response phase. Sample members who 
completed the interview during the early response phase received either $30 (first follow-up 
interview respondents) or $50 (first follow-up interview nonrespondents). 

On average, the BPS:04/09 interview took 19.6 minutes to complete. Overall, web 
interviews were significantly shorter at 19.4 minutes than telephone interviews were at 20.2 minutes, 
but not significantly shorter than field interviews were at 19.6 minutes. The telephone interviews 
were not significantly longer than the field interviews. Average section completion times were 
1.3 minutes for the Front End, 4.1 minutes for Enrollment History, 4.5 minutes for Enrollment 
Characteristics, 6.1 minutes for Employment, and 4.1 minutes for Background. On average, the 
BPS:04/09 abbreviated interview took 10.5 minutes. 

An evaluation of the quality of the data provided by the BPS:04/ 09 student interview 
showed that methodological features built into the instmment as well as training and supervision of 
interviewing staff aided in the successful administration of the interview. The design of assisted 
coding systems in the instrument and the training of interviewers on coders appeared successful. 
Overall, expert coders agreed with major, occupation, and industry codes chosen in the interview 
92 percent of the time, recoded codes chosen to a new value about 7 percent of the time, and were 
unable to choose a code based on too vague a text string about 1 percent of the time. Only 3 percent 
of text strings provided in the interview lacked a code and required upcoding. The appearance of 
help text and conversion text in the instmment also appeared to improve question response. Help 
text was accessed significantly more often during interviews by telephone and field interviewers than 
by web respondents. It should be noted that interviewers had been encouraged to use help text, as 
needed, as this feature of the instmment was emphasized during telephone interviewer training. 
Seventy-six percent of the cases where conversion text was triggered in the interview were converted 
to a response after the conversion text was displayed. Overall, the item-level nonresponse analysis 
yielded 19 items out of 385 interview items with more than 5 percent missing data. 

Debriefing of tracers and field interviewers at the end of data collection indicated that 
frequent monitoring of telephone interviewers, a help desk that tracked and resolved difficulties 
encountered by sample members attempting to complete the web interview, and quality circle 
training and feedback meetings were useful as data collection QC procedures. Most interviewers 
indicated that they felt they had all the tools necessary to successfully administer the BPS:04/ 09 
student interview. 
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Chapter 4. 

Transcript Data Collection, Outcomes, 

and Evaluation 


For the first time in the BPS series of studies, postsecondary transcripts were collected for a 
BPS cohort. As part of BPS:04/ 09, transcripts from all postsecondary institutions attended between 
July 1, 2003 and June 30, 2009 were requested for the BPS:04 cohort. To ease burden on 
participating institutions, the BPS:04/ 09 transcript collection was combined with the transcript 
collection for the 2008 / 09 Baccalaureate and Beyond Longitudinal Study under the 2009 
Postsecondary Education Transcript Study (PETS:09). This chapter provides an overview of the 
BPS:04 portion of the transcript collection and will describe the processes and systems developed 
for collecting transcripts. It will also report on transcript keying and coding activities and the data 
and process evaluation procedures introduced to assure data quality. 

4.1 Transcript Data Collection and Response Rates 

A Transcript Control System (TCS) was designed to manage the transcript and other 
institution data requested from institutions attended by the BPS:04 cohort. Institution contactors 
(ICs) served as liaisons to institutions that provided the requested materials through a variety of 
possible submission methods, including a study website. Transcripts for a total of 16,960 students 
were received. The details of transcript data collection and response rates are included in this 
section. 

4.1.1 Transcript Control System 

The integrated, web-based TCS supported each step of the BPS:04/ 09 transcript collection, 
including project management, communications, and tracking. The TCS comprised several transcript 
management systems: the Institution Contacting System was used to store and access data on students 
and track efforts to obtain their transcripts; the Data Receipt System managed data received on sample 
members, including transcripts and catalogs for the institutions attended; and the Keying and Coding 
System (KCS) facilitated the efficient and secure capture of data from student transcripts. See section 

4.2 for a detailed discussion of the development and use of the KCS. Transcript control system data 
were stored in SQL databases for ready access in reporting, documenting and delivering transcript 
data. 

4.1.2 Training of Institution Contacting Staff 

Institution contacting staff consisted of eight ICs and two quality control supervisors (QCS) 
who were responsible for staff supervision. Prior to the start of transcript data collection, the ICs 
were trained over a 2-day period on transcript and catalog collection, gaining cooperation, and 
problem resolution. Training included information on BPS and a review of confidentiality 
regulations. Activities focused on guidelines for interactions with institution staff, gaining 
cooperation, collection of catalogs and transcripts, and collection and receipt systems. During the 
transcript collection period, staff were briefed on their progress, asked questions, and discussed 
issues at weekly quality circle meetings. The IC training agenda is included in appendix G. 
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4.1.3 Transcript Collection Procedures 

Transcripts were requested from the 3,030 eligible postsecondary institutions attended by the 
18,640 BPS:04/09 sample members, including each sample member’s base-year. National 
Postsecondary Student Aid Study (NPSAS) institution, and any additional institutions attended as 
reported in the first follow-up (BPS:04/ 06) and second follow-up (BPS:04/ 09) student interviews or 
noted on other transcripts collected during the study. In addition, if an institution had copies of 
transcripts received from any transfer schools attended by the BPS sample member, the transfer 
transcripts were requested as well. 

Transcript data collection comprised two phases. Phase 1 began in November 2008 
collecting transcripts requested from each student’s NPSAS institution and from any additional 
institutions students reported attending in the first follow-up (BPS:04/ 06) student interview. Phase 2 
began in October 2009 after completion of the second follow-up (BPS:04/ 09) student interview. 
Phase 2 transcripts were requested based on additional enrollment reported in that interview. 
Additionally, in early 2010, transcript requests were made of any transfer institutions identified on 
collected transcripts. In the phase 2 requests, institutions were also asked to provide updates for 
incomplete grades noted on previously collected transcripts. 

At the start of the transcript collection, a transcript request packet was sent to the director of 
the institutional research office at each institution. In the absence of an office of institutional 
research, packets were sent to the registrar’s office. The packet contained notification materials for 
transcript data collection (appendix H), including the following: 

• a letter introducing PETS:09, 

• an introductory letter from the National Center for Education Statistics (NCES); 

• a letter of endorsement from the American Association of Collegiate Registrars and 
Admission Officers (AACRAO); 

• a list of other endorsing agencies; 

• information regarding how to log on to the study’s secure website; 

• descriptions of and instmctions for the various methods of providing transcripts; and 

• excerpts from the Family Educational Rights and Privacy Act (FERPA) that illustrated 
the transcript collection’s compliance with the legislation. 

Follow-up calls by trained ICs were placed 2 days after the initial mailing to ensure receipt of 
the packet and to answer any questions about the study. Prompting calls were made and reminder e- 
mails sent, as needed, from November 2008 through February 2010. 

Transcript submission. Institutions were provided the following seven options for 
submitting transcripts: 

1. File upload to the study ivehsite. Institutions were asked to submit electronic transcript files, 
preferably in an extensible markup language (XML) or electronic data interchange (EDI) 
format that conformed to the Postsecondary Electronic Standards Council standard. If 
the transcript data were not already in one of the two preferred formats, the institution 
was asked to convert the files before loading, or to prepare files using the file 
specifications provided on the study website. The transcript files were submitted directly 
to the secure study website. The latest technology systems were incorporated into the 
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transcript website application to ensure strict adherence to NCES confidentiality 
guidelines. The web server included a Secure Sockets Layer (SSL) encryption certificate 
and was configured to force encrypted data transmission over the Internet. All of the 
data entry modules on the site were password protected, and the user was automatically 
logged out of the system after 20 minutes of inactivity. Just as with all the submission 
methods, once the transcript files were received, they were immediately moved to a 
secure project folder accessible only to a subset of project staff. 

2. Submission of electronic transcripts by secure file transfer protocol (FTP) server. Transcript files could 
be submitted using an FTP server, which ensured an encrypted control session. As with 
the file upload, it was preferable for files to be submitted using an XML or EDI format, 
but files could be submitted in virtually any file layout. After being copied to the secure 
project folder, the files were immediately deleted from the FTP server. 

3. Submission of transcripts via eSCRIP-SAFE. eSCRIP-SAFE is a third-party vendor that 
receives and electronically converts transcripts to PDF files, then stores them on a secure 
server. Institutions registered with this service send data by secure internet connection to 
the eSCRIP-SAFE server, where they can be downloaded only by a designated user. The 
electronic transcript files downloaded by project staff from eSCRIP-SAFE were saved 
only to the secure project folder. 

4. Submission of transcript files as encrypted attachments to e-mail. Electronic transcript files could 
be emailed as attachments to the project e-mail account. Guidelines on encryption and 
creating strong passwords for transcript attachments were provided to the institutions. 
Encrypted transcript files were moved to the secure project folder and deleted from the 
e-mail folder immediately. 

5. Submission of transcript files through a dedicated server at the University of Texas at Austin. A 
dedicated server at the University of Texas at Austin, developed to allow transcript 
exchange among registered institutions, was provided as an option to institutions 
submitting transcripts to the study. The server supported both XML and EDI formats. 

6. Submission of transcripts via secure electronic fax. Transcripts were accepted via secure 
electronic fax. To safeguard against information being misdirected or intercepted by 
individuals for whom access was not intended or authorized, RTI protocol only allowed 
for transcripts to be sent to an e-fax server housed in a secured data center at RTI. The 
transcript data were stored on the server as electronic (PDF) files. To ensure 
confidentiality, institutions were asked to send a test fax with nonsensitive data and to 
use a specific fax cover sheet from the project website that included a confidentiality 
statement. After being received and copied to the secure project folder, transcript files 
were deleted from the e-fax server. 

7. Submission of transcripts via FedEx. Transcripts were accepted via FedEx when none of the 
other methods was manageable. To safeguard confidentiality, institution staff were 
instmcted to redact any personally identifiable information from the transcript including 
student name, address, date of birth, and Social Security Number (if present). Paper 
transcripts were kept in a locked file cabinet in RTFs secure data receipt facility, to which 
only a limited number of BPS:04/09 transcript staff had access. After the paper 
transcripts were scanned and stored electronically, they were shredded. 


BPS:04/09 Full-scale Methodology Report 


53 



Chapter 4. Transcript Data Collection , Outcomes , and Evaluation 


In addition to transcripts, other information from each institution was needed for keying and 
coding. Institutions were asked to provide academic calendar and grading system information on the 
study website. If course catalogs could not be obtained separately through institution websites or 
through CollegeSource Online, a resource for over 50,000 postsecondary institution catalogs, they 
were requested from institutions. 

Transcripts and course catalogs received were inventoried, assigned unique identifiers, 
reviewed for any problems with legibility and completeness, and logged each day in the data receipt 
system. Project staff used daily monitoring reports to review problem transcripts and ICs assisted 
with resolving transcript problems directly with institutions. 

4.1.4 Institution Website 

The PETS:09 website (figure 13) was the portal used to collect institution data and 
transcripts. The website contained information about PETS, including research topics, the transcript 
collection, how transcript data would be used, answers to frequently asked questions, and 
confidentiality assurances. Contact information for the transcript data collection help desk and 
project staff at RTI, as well as links to the main NCES and RTI websites, were also included on the 
home page. From the secure portion of the website, institutions could view the list of their sampled 
students, view detailed instructions for providing transcript data, and upload data. 

Various systems were incorporated into the website application to ensure strict adherence to 
NCES confidentiality guidelines, including the following: 

• a Secure Sockets Layer (SSL) Certificate ensured secure data transmission over the 
Internet, 

• all data entry modules were password protected, 

• users were automatically logged out of the system after 20 minutes of inactivity, and 

• files uploaded to the secure website were immediately moved to a secure project folder 
accessible only to a subset of project staff. 
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Figure 13. Institution website home page: 2009 



Home About B&B and BPS Endorsements FAQs Leoal Authority 


Contact Us Help Login 


We are now collecting transcripts for enrollment reported by sampled students in a recent 2009 student interview. 

If you were asked to provide transcripts for PETS earlier in 2009, your institution is being recontacted for one or both of 
the following reasons: (1) to provide transcripts for new students who had not reported attending your institution 
previously or (2) to provide transcripts for students who reported new enrollment at your institution. 

Thank you for your participation in this important study. 


Home 

Welcome to the 2009 Postsecondary Education Transcript Study (PETS:09) institution website The National Center for Education 
Statistics (NCES) is currently conducting two postsecondary longitudinal studies, the 2008/09 Baccalaureate and Beyond 
Longitudinal Study (B&B:08/09) and the 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). PETS:09 is the 
transcript collection component for these two studies. 

Approximately 3,000 institutions will be included in PETS:09. Across both studies, we expect to collect transcripts for approximately 
45,000 postsecondary students. 

To enter information for your institution on this secure website, log in using your institution's IPEDS UNITID and password, 
which are printed on your transcript reguest letter. You may also contact the Help Desk . 

Log in to complete the following key tasks: 

• Complete the Institution Information Page. 

• Complete the Transcript Data Transmission Page 

• Transmit sampled students' transcript data. 

• Upload CIP code mappings. 

• View detailed instructions for providing transcripts. 

• View/Print a list of Sampled Students. 

Through this website, you can 

• Get answers to frequently asked questions fFAQst about the transcript collection. 

• Visit the National Center for Education Statistics website for more information on the B&B and BPS studies. 

• Find examples of research reports prepared using PETS data. 

• View a Disclosure Notice ^9KBt that may be printed and placed in each sampled students file indicating that data have 
been provided to the study. 

• Review the list ot organizations endorsing B&B and BPS. 

• Contact the Help Desk for assistance. 

If you need assistance, send an e-mail to PETS@rti.org or call the Help Deskat 1-877-256-8029 for questions. We also welcome 
your comments and suggestions for improving the study and this website. 



VERIFY ► 


PETS:09 is the transcript collection component of the following studies. 

Baccalaureate and Beyond Longitudinal Study 
OMB clearance No:1850-0729 Exp.Date:02/28/2011 



Beginning Postsecondary Students Longitudinal Study 
OMB clearance No:1850-0631 Exp.Date:04/30/2011 



National Ce-ter for Education Statistics. ["sttute of Education Sciences 
U.S. Department of Education 
1990 K Street, NW, Washington, DC 20006 USA 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 
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4.1.5 Transcript Collection Response Rates 

Institution-level transcript collection. Table 26 provides institution participation rates by 
institution type. The institution sample for the transcript collection included 3,100 distinct 
institutions attended by the BPS:04 cohort. Of the 3,100 institutions, it was determined that 2 
percent were ineligible because the institution had closed or because a sample member had enrolled 
in, but never actually attended the institution. Of the remaining 3,030 institutions, 2,620 (87 percent) 
provided transcripts for the cohort. Across the institution types represented, participation in the 
transcript collection ranged from 71 percent at the private for-profit less-than-2-year institutions, to 
93 percent at public 4-year doctorate -granting institutions. 


Table 26. Eligible institution participation, by institution type: 2009 


Institution type 

Total eligible 

institutions 

Institution-level participation 1 
Number Percent 

Total 

3,030 

2,620 

86.6 

Public 

Less-than-2-year 

70 

50 

77.9 

2-year 

920 

810 

88.9 

4-year non-doctorate-granting 

300 

270 

90.4 

4-year doctorate-granting 

260 

240 

93.4 

Private nonprofit 

2-year-or-less 

90 

80 

85.9 

4-year non-doctorate-granting 

510 

460 

91.3 

4-year doctorate granting 

240 

210 

89.0 

Private for-profit 

Less-than-2-year 

260 

180 

70.5 

2 years or more 

390 

310 

78.4 


1 An institution was considered a participant if it provided a transcript for at least one student. Sixteen of the 
participating institutions are not represented in the institution type rows due to unknown institution type. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary 
Student Aid Study (NPSAS:04) and 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). 

As shown in table 27, institutions preferred submitting transcripts via secure fax, with about 
69 percent using that method. Common reasons cited by institutions for not participating in PETS 
included lack of available staff to handle the request for transcripts and the timing of the transcript 
request. 
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Table 27. Institution transmission mode for transcript data: 2009 


Transmission mode 

Institutions 


Number 

Percent 

Total 

2,620 

100.0 

Electronic fax 

1,800 

68.5 

Upload to study website 

470 

17.8 

Federal Express 

170 

6.5 

E-mail 

120 

4.5 

Secure FTP 

50 

1.9 

eSCRIP-SAFE™ 

20 

0.6 

University of Texas server 

# 

0.1 


# Rounds to zero. 

NOTE: Detail may not sum to totals because of rounding. FTP = file transfer protocol. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary 
Student Aid Study (NPSAS:04) and 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). 

Transcript-level collection. Transcripts were requested for each of the BPS:04 sample 
members from each of the known institutions they attended. At the transcript level, 25,120 (87 
percent) of the requested transcripts were received, keyed, and coded. Additional transcripts were 
collected but not keyed and coded if they only contained graduate coursework, were illegible, were 
duplicates of received transcripts, or were transcripts for students learned to be deceased. 

During phase 1, transcripts were requested from each student’s NPSAS institution as well as 
from any additional institutions students reported attending in the first follow-up (BPS:04/ 06) 
student interview. Eighty-eight percent of the phase 1 transcripts were received. During phase 2, 
transcripts were requested based on additional enrollment reported in the second follow-up 
(BPS:04/ 09) interview as well as any transfer institutions identified on collected transcripts. Eighty- 
five percent of the phase 2 transcripts were received. In the phase 2 requests, institution staff 
members were also asked to provide updates to incomplete grades noted on previously-collected 
transcripts. In total, 575 courses with incomplete grades were identified and 173 courses (30 percent) 
were replaced with an updated grade. 

Student-level transcript collection. At the student level, a transcript was received from at 
least one institution for 16,960 sample members (92 percent), and a transcript was received from the 
sample member’s NPSAS institution, the first postsecondary institution attended, for 16,540 sample 
members (89 percent). A transcript was received from each institution attended for 16,110 sample 
members (87 percent). Table 28 shows the transcript collection results at the student level. 


Table 28. Student-level transcript collection results: 2009 


Student sample 

Number 

Percent 

Total 

18,640 

100.0 

Transcript respondents 1 

16,960 

91.5 

NPSAS transcript received 

16,540 

89.2 

All student's transcripts received 

16,110 

86.9 

Transcript nonrespondents 

1,580 

8.5 


A student was considered a transcript respondent if we received a transcript from one or more institutions. 
NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary 
Student Aid Study (NPSAS:04) and 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). 
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4.2 Transcript Keying and Coding 

Keying and coding of transcripts was completed using a data entry application. Several 
quality control activities evaluated the various types of data collected (e.g. courses, major/ field of 
study). This section will describe keying and coding procedures as well as outcomes. 


4.2.1 Transcript Keying and Coding Procedures 

Transcript keying and coding was performed by a team of specially trained, data entry 
(keyer/ coder) staff using a web-based data entry application. Work was evaluated using several 
quality control activities designed for various types of data collected (e.g., courses, major/ field of 
study). The keying and coding sections (figure 14) collected the following transcript information (for 
a full list of keyed and coded transcript data elements, see appendix I): 

• Case information. Preliminary transcript information including student name, address, and 
high school graduation date. 

• Schools and terms. Names of the transcript school and any transfer institutions reported on 
the transcript, terms attended at these schools, and attempted and accepted transfer 
credits. 

• Academics. Academic honors awarded (e.g.. Dean’s or President’s List) and any 
probations, by term. 

• Tests. Institution exams (e.g., competency and placement exams) or externally 
administered exams (e.g., SAT and GRE), and related scores. 

• Degrees and majors. Degree programs attempted or earned, such as a bachelor’s or 
associate’s degree, degree receipt dates, and honors awarded at graduation such as cum 
laude. The specific majors or fields of study for each degree were coded in this section. 

• Courses. Key data on courses listed on transcripts, including the terms in which the 
courses were taken, course numbers and names, and grades and credit or clock hours 
earned. In this section, each course was also coded for standardization. 


Figure 14. Keying and coding system sections: 2009 


i Case information 

Schools and terms 

1 

Student name 

School name, city, state 

1 

Address 

Attendance dates by term 

1 

1 

1 

High school graduation date 

Transfer credits attempted and accepted 

1 

1 

Academics 

Tests 

1 

Honors 

Exam type 

1 

1 

Probations 

Date taken 

1 

1 

1 

Degrees and majors 

Courses 

1 

1 

Program (e.g., BA, AA) 

Course number 

1 

Degree name 

Course name 

1 

Date received 

Course code 

1 

1 

Graduation honors 

Course credits 

1 

1 

Major 1 , 2; Minor 1 , 2; Concentration 

Grade 



SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 
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To help to ensure the quality of data keyed and coded, specific features were incorporated 
into the KCS. For example, the KCS provided links to institution course catalogs for easy reference; 
limited ranges and the types of characters input for fields such as dates and exam scores; and 
required that postsecondary institutions, majors, and courses be coded using specially designed 
coders. 

KCS coding systems. The school and major or field of study coders incorporated into the 
KCS were assisted coders identical to those used in the BPS:04/ 09 student interview instmment. 

The school coder used the set of institutions contained in the Integrated Postsecondary Education 
Data System (IPEDS), developed by the National Center for Education Statistics (NCES) 

( http: / /nces.ed.gov/IPEDS/) . The major or field of study coder was based on the 2010 
Classification of Instructional Programs (CIP) taxonomy developed by NCES 
(http:/ /nces.ed.gov/ipeds/cipcodel . CIP codes not associated with postsecondary majors or fields 
of study were removed from the coder, including those for basic skills and developmental education, 
citizenship activities, health-related knowledge and skills, interpersonal and social skills, leisure and 
recreational activities, personal awareness and self-improvement, and high school/ secondary 
diplomas and certificates. 

The KCS course coder was similar to the KCS school and major or field of study coders, 
with the addition of certain search features. When a text string with the course title was entered, a 
keyword search based on the course title was conducted on the underlying database allowing the 
keyer/ coder staff person to select the best option from a list of possible course options returned. If 
the course title did not adequately capture the description of the course in the institution catalog, 
keyer/ coders could search the course coder database using keywords found in the course 
description in the institution course catalog, or they could do searches by broad categories and by 
database codes. The KCS also included a feature for entering problem sheets for particular schools 
or transcripts. Problem sheets were categorized and routed to supervising staff for resolution. 

Development of the PETS course coder. The underlying database for the course coder in 
the KCS included 2,119 course codes and code definitions. Course codes were developed by 
integrating selected courses from the College Course Map (CCM) (Adelman, C. 2004) into the 2010 
CIP taxonomy from NCES. PETS codes were represented by six digits in keeping with the CIP 
taxonomy: the first 2-digits indicated the most general category; the first 4-digits narrowed the focus 
to a subcategory; and the complete 6-digit code provided the most specific definition of the subject. 
Figure 15 provides a visual representation of the structure of the codes. 

Figure 15. CIP code diagram: 2009 

2-digit 

general category"! 

0 1 0 6 

4-digit 

subcategory 

6-digit 

specific definition of subject 

NOTE: CIP = Classification of Instructional Programs. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 
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Course codes in the CCM taxonomy, which used the same 6-digit structure as the then- 
current CIP, were developed through extensive transcript analysis and with input from expert 
advisors, including postsecondary faculty familiar with the fields of study. To create a 
comprehensive course coder for PETS:09, content from the CCM was incorporated into the 2010 
CIP in two ways: (1) course codes found in the CCM without equivalents in the CIP were added as 
new codes, and (2) CCM codes with equivalent CIP codes were reviewed and, when additional 
details or examples were found, they were added to the CIP definitions. The first method resulted in 
the addition of 352 unique CCM course codes to the KCS course coder. These additions were 
placed alongside related topics in the 2-digit category and 4-digit subcategory stmcture common to 
both the CCM and CIP taxonomies. To make these additions easily identifiable, the last 2 digits in 
their codes used a unique numbering scheme, starting with 98 and descending as needed. An 
example can be seen in code 01.0698 in figure 16. 

The second method for integrating the CCM and the 2010 CIP resulted in adding content to 
the definitions of 316 CIP codes. The additional text increased the likelihood of identifying 
appropriate course codes using the keyword searchable KCS course coder. In instances where a CIP 
code was elaborated, the CCM content was placed at the end of the CIP definition. Finally, in 
addition to content from the CCM, 47 general and other codes were added to the KCS course coder 
using KCS course coder fifth and sixth digit values of “00” for general and “99” for other, when these 
codes were not already present in the CIP. Figure 1 6 illustrates a representative set of codes in the 
KCS course coder. 


Figure 16. Course codes layout in the KCS coders: 2009 


01 ) Agriculture, agriculture operations, and related sciences. 

01.06) Applied Horticulture and Horticultural Business Services. 
01.0601) Applied Horticulture/Horticulture Operations, General. 

01.0603) Ornamental Horticulture. 

01.0604) Greenhouse Operations and Management. 

01.0605) Landscaping and Groundskeeping. 

01.0606) Plant Nursery Operations and Management. 

01.0607) Turf and Turfgrass Management. 

01.0608) Floriculture/Floristry Operations and Management. 
01.0698) House Plants, Indoor Plants and/or Herbaceous 

Ornamental Plants. 


A program that focuses on the general production and 
processing of domesticated plants, shrubs, flowers, foliage, 
trees, groundcovers, and related plant materials; the 
management of technical and business operations 
connected with horticultural services: and the basic scientific 
principles needed to understand plants and their 
management and care. Also organic gardening and/or 
arboriculture. 


Any applied horticulture and horticultural business services 
course that deals with topics of house plants, indoor plants, 
and/or herbaceous ornamental plants 


01.0699) Applied Horticulture/Horticultural Business Services, Other. 


NOTE: Bold text represents content and codes added from the College Course Map taxonomy (CCM). KCS = keying and coding 
system. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


Transcript courses were originally coded using the available draft of the 2010 CIP. The 
database for the course coder in the KCS was updated when the final version of the 2010 CIP was 
released in July 2009. Compared to the draft version, the final 2010 CIP included 80 new codes, 21 
codes with different code numbers, and two codes that were deleted from the draft. For the 80 new 
codes in the final 2010 CIP, courses on transcripts previously coded with similar codes from the 
draft 2010 CIP were reviewed by keyer/ coder staff to determine if they fit better into the new CIP 
codes. There were 17,778 courses previously coded with draft 2010 CIP codes that were updated to 
the final 2010 CIP codes. There were just three courses previously coded with deleted CIP codes 
from the draft 2010 CIP which, when reviewed, were assigned to similar, related codes in the final 
version of the 2010 CIP. 
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4.2.2 Training of Transcript Keyer/Coder Staff 

Over the course of three separate 5-day trainings, beginning in January 2009, 71 keyer/ coder 
staff were trained to use the KCS. Keyer/ coders were supervised by five QCS who were responsible 
for administrative and management issues, as well as quality review of keyed and coded transcripts 
and keying and coding, as needed. 

Each training session began with background on BPS, review of confidentiality regulations, 
fingerprinting, and signing of notarized affidavits. These activities were followed by an overview and 
discussion of the different types of transcript formats and key data elements to be located and 
entered into the KCS. Presentations on keying and coding fundamentals were followed by problem- 
solving exercises and practice sessions. The fifth day of training consisted primarily of supervised 
keying and coding practice using actual transcripts, followed by a practicum exam on which all 
trainees were required to obtain 90 percent or better proficiency for certification. The training 
agenda for transcript data collection is included in appendix G. 

In conjunction with the above trainings and because of the wide variation in transcript 
layouts and information provided by the institutions, quality circle meetings were held weekly for the 
first 8 months of keying/ coding. During the meetings, QCS and a group of keyer/ coders were 
briefed on production and performance measures and were invited to ask questions or raise 
concerns. Topics discussed ranged from the use of specific CIP codes, to keying of the more 
problematic data elements. As the project progressed, the frequency of the quality circle meetings 
was adjusted to biweekly. 

Often as a result of feedback during quality circle meetings, additional, narrowly focused 
trainings were held as needed for specific topics, such as entering multiple transcripts for sample 
members, coding of electronic transcripts, and the use of problem sheets to record issues or 
questions with transcripts. 

4.2.3 Transcript Keying and Coding Outcomes 

Keying and coding was performed on 25,120 transcripts for 16,960 students. These 
transcripts included a total of 583,380 courses, 36,680 terms, and 24,230 degrees. Of the 583,380 
courses coded, 75 percent were coded with a specific 6-digit code that was neither a general nor an 
other code. General codes were selected for 23 percent of the courses coded and were typically chosen 
for the many introductory level courses, while other codes were selected for 2 percent of the courses 
when no more specific code matches in the KCS course coder were found for a course on a 
transcript. Uncodeable courses accounted for 1 percent of all courses, often due to unclear course 
titles or inadequate information on course content. The results of course coding are shown in figure 
17. 
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Figure 17. Course coding results: 2009 
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Source: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

4.2.4 Evaluation of Transcript Keying and Coding 

Multiple evaluation steps were taken to ensure the quality of transcript data entered into the 
KCS. These evaluation activities included rekeying a sample of data elements, expert coding a 
sample of course and major/ field of study coding, and upcoding of text strings for institutions or 
other transcript data elements that could not be coded initially. 

Rekeying. To evaluate the reliability of transcript data keyed into the KCS, approximately 
10 percent (2,670) of the transcripts were randomly selected to be rekeyed. A subset of transcript 
data elements were rekeyed by quality control supervisors, which took approximately 1 0 to 15 
minutes per transcript, depending on the number of the selected data elements found on the 
transcript (e.g. the number of terms attended). Figure 18 shows agreement rates for the rekeying 
activity, organized by keying and coding section. For both rekeying and recoding activities, the 
Cohen’s Kappa statistic was used to assess inter-rater reliability between the original coder and 
quality control supervisors, or expert coders. Cohen’s Kappa measures the proportion of agreement 
between raters, above what would be expected by chance. A kappa score of 0.81-1.00 is considered 
“near perfect agreement,” 0.61-0.80 is “substantial agreement,” and 0.41-0.60 is “moderate 
agreement.” All of the rekeyed items had at least moderate agreement, with case information, terms, 
and degrees all within the range of near perfect agreement. 
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Figure 18. Rekey kappa values, by transcript section: 2009 

Kappa 



Case information Schools Terms Tests Degrees 

Transcript section 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


Data collected in the test section of the KCS (exam name, date taken, and score), however, 
had a noticeably lower value for kappa (0.42) than other data elements. Further investigation into 
test data on transcripts revealed that of the 2,620 schools that provided transcripts, only 23 schools 
included Advanced Placement tests with scores on their transcripts. Instead, tests, particularly exams 
for which course credit is awarded, are often included on transcripts in a format more similar to 
courses (e.g. “AP biology, 3.0 credits”). Due to the low interrater reliability score and frequency with 
which test data was found on transcripts, this category of data was determined to be unreliable and 
will not be included in BPS data file because its presence on postsecondary transcripts was 
determined to be unreliable. 

Expert coding. Expert coding was performed on 47,428 courses, both to evaluate the 
reliability of coded data and to create feedback opportunities to improve data quality. Expert coding 
used more experienced (expert) staff, all of whom held at least a bachelor’s degree, to recode a 
subsample of coded courses and to provide feedback to keyer/ coders on course code selection. 
Expert coding was performed from the beginning of the keying and coding process and continued 
until its conclusion so that keyer/ coder staff could receive feedback on their performance and 
additional training needs could be addressed properly. 

Initially, the expert coding process included two steps. In expert coding 1 (EC1), expert 
coder staff reviewed course information and selected a code, which was then compared to the 
keyer/ coder’s choice. In cases where the keyer/ coder and expert coder selections did not match, 
expert coding 2 (EC2) was performed to assess the reliability of EC1. EC2, in addition to being 
performed on all cases where EC1 and keyer/ coder choices did not match, was also performed on a 
15 percent random sample of codes where the EC1 and keyer/ coder agreed. 

EC2 was performed as a review of the quality of the EC1 staff and included review of course 
information and deciding upon the EC1 choice, the keyer/ coder choice, or an entirely different 
code — to avoid potential bias, the EC2 could not identify which selection was made by the EC1 or 
keyer/ coder. EC2 review of keyer/ coder EC1 disagreements added reliability to the EC1 code 
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selections upon which keyer/ coder feedback was developed. EC2 was performed by the same 
project staff responsible for keyer/ coder training and course code development. 

Based upon a sample of 1,330 disagreements between keyer/ coder and EC1, EC2 agreed 
with EC1 in 58 percent of the cases, with the keyer/ coder in 33 percent of the cases, and selected a 
different code (neither the EC1 nor the keyer/ coder’s choice) in 9 percent of the cases. The EC2 
staff agreed with expert coder choices significantly more than the keyer/ coder choices: 

X 2 (1,IV= 1,330) = 90.04,/> < 0.01). 

As with keying and coding, expert coding was performed in batches by school. Courses were 
not expert coded until all of a school’s transcripts had been keyed and coded. For both EC1 and 
EC2, expert coders reviewed course number and name and had access to course catalogs to make 
coding decisions. 

For the purpose of reviewing keyer/ coder work and providing feedback, expert coding was 
performed on both random and cluster samples of courses. EC1 was performed on a random 10 
percent sample of all courses from each school. For schools with fewer than 10 total courses, all 
courses were expert coded. Cluster sampling was used to select courses coded with other codes (e.g. 
26.0299, biochemistry, biophysics and molecular biology, other). Courses coded as needs review or 
uncodeable were also reviewed in expert coding. 

Interrater reliability for course coding was assessed using 5,000 courses randomly selected 
for calculating agreement statistics. Cohen’s Kappa statistic was used to assess interrater reliability 
between the original coder and expert coders. Expert coding results are shown in figure 19. 
Agreement rates are shown at three levels of specificity: 2-digit, 4-digit, and 6-digit. At the 2-digit 
level, the kappa statistic indicates near-perfect agreement between keyer/ coder and expert coder. At 
both the 4- and 6-digit levels, the kappa statistic indicates substantial agreement. 

Figure 19. Expert coding results, kappa by level of specificity: 2009 

Kappa 
1.00 - 
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0.40 - 
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2-digit 4-digit 6-digit 

Level of specificity 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

Review of “other” courses. For the first six months of keying, courses coded using the 
“other” category in the PETS:09 coder were reviewed by expert coders with the goal of minimizing 
the use of the category. Keyer/ coder training emphasized that “other” codes should be reserved for 
courses that fit within a 4-digit subject area but for which more specific 6-digit codes in that series 
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were not appropriate. “Other” was not intended for coding problematic courses or those for which 
additional analysis would result in a more accurate code. Expert coders provided direct feedback to 
keyer/ coders on cases for which there were more appropriate coding choices. 

In addition to the expert coding performed, “other” codes were also reviewed to determine 
if additional codes should be added to the PETS coder. The PETS course taxonomy included 231 
courses with an “other” designation, such as 31.0599, “Health and Physical Education/Fitness, 
Other,” or 23.9999, “English Language and Literature/Letters, Other.” A review of the courses 
coded as “other” was undertaken to determine if there were common subjects within the codes that 
would merit introduction of new codes. The median number of uses of “other” for all course codes 
was 132. This median was used as the threshold for adding a new code: if 132 instances of the same 
subject could be identified within the courses coded using the “other” code, a new code would be 
added. However, a review of “other” codes did not identify any subjects that met this threshold, so 
no new codes were added. 

Upcoding and reliability recoding for major/ field of study. In a process known as 
upcoding, text strings for 783 entries for field of study that were not coded by keyer/ coders were later 
reviewed by project staff to determine if an appropriate code could be identified. For the uncoded 
majors, project staff was able to identify a major code in 38 percent of the cases. In 50 percent of 
the uncoded cases, data on the transcripts were too vague to identify an appropriate code and, in 12 
percent of cases, the original code selection of “other” was correct. In addition to this upcoding 
activity, a random sample of 3,932 coded majors was recoded as needed as a key-rekey step to evaluate 
the reliability of the field of study data. For these cases, the coder and recoder agreed in 86 percent 
of the cases. The results are shown in figure 20. 

Figure 20. Major/field of study upcoding: 2009 


Percent 



Original was Original was Original was too Original was Original was too Upcodedfrom 

correct recoded vaguetocode correct vaguetocode missing 


Random sample for reliability evaluation Major was not coded or coded as "other" 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09) 

Upcoding for institutions and variables with “other, specify” options. Uncoded text 
strings for institutions were upcoded through a review by project staff to determine if an appropriate 
code could be identified. This task was performed on 701 cases by staff with greater familiarity with 
postsecondary institutions and with additional resources for researching school names and locations. 
The results of this institution upcoding are shown in figure 21. In 55 percent of the cases, the 


BPS.04/09 Full-scale Methodology Report 


65 


Chapter 4. Transcript Data Collection, Outcomes, and Evaluation 


institution could not be identified in the Integrated Postsecondary Education Data System (IPEDS), 
and thus remained uncodeable. Analysts were able to code 43 percent of the previously uncodeable 
institutions while, for 2 percent of cases, the school name could not be identified at all. 

Figure 21. Institution IPEDS upcoding: 2009 



NOTE: Detail may not sum to totals because of rounding. 

Source: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09) 

In addition to institutions, transcript data elements with “other, specify” options included: 

• non-course credits awarded (e.g. course credit for Advanced Placement tests), 

• tests (e.g., SAT), 

• academic honors (e.g.. Dean’s List), 

• academic probations (e.g., academic probation), 

• degree programs (e.g., associate’s), 

• grades (e.g., R), 

• bachelor’s degree types (e.g., Bachelor of Education), and 

• graduation honors (e.g., with distinction). 

All items coded as “other, specify” were reviewed by analysts to determine if the text strings 
could fit into existing choices or if there were common strings that merited addition of a new 
choice. For example. Bachelors of Education was not included in a drop-down menu for bachelor 
type, therefore it was entered as a text string under “Other, specify.” When the value appeared 
repeatedly as a text string, it was assigned as a category and upcoded accordingly. Table 29 shows the 
results of “other, specify” upcoding. The total number of cases is shown for each data element along 
with the number and percent that were upcoded. 
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Table 29. Upcoding of "other, specify" data: 2009 


Data elements with "other, specify" option 

Number of "other, 
specify" cases 

Number 

upcoded 

Percent 

upcoded 

Non-course credit awarded 

3,370 

2,520 

74.6 

Tests 

3,840 

1,590 

41.5 

Term honors 

22,350 

14,810 

66.3 

Term probations 

7,250 

6,670 

92.0 

Degree program 

170 

40 

22.4 

Grade 

32,630 

18,610 

57.1 

Bachelor's degree 

430 

310 

71.8 

Degree honors 

270 

170 

61.9 


NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


Keyer/ coder staff debriefing. Near the conclusion of keying and coding, a debriefing 
focus group was held with 7 keyer/ coders who had collectively keyed and coded more than 1 0,000 
transcripts. Two participants had also performed duties as QCS. Focus group participants agreed 
that the keyer/ coder training had been helpful and prepared them for the task. They also found 
quality circle notes and meetings to be useful. The keying and coding system facilitated entry of 
transcript data, although some data elements in the system were not commonly found on transcripts, 
such as “transfer credits for GPA” and “state basic skills tests.” 

Regarding course coding, focus group participants found the search features of the KCS to 
be useful, but certain course subjects were difficult to code, such as engineering and computer 
courses, as well as some education courses, when it was difficult to discern whether the course was 
about learning the topic itself or learning about how to teach the topic. Finally, focus group 
participants indicated that identifying remedial courses was sometimes difficult, for instance when 
the course description sounded like it could be remedial but without stating so explicitly. In such 
cases, keyer/ coders were sometimes able to confirm a course was remedial by noting a grade greater 
than an F with no credits awarded. 

4.3 Timing of Transcript Keying and Coding 

Transcript keying and coding was conducted from January 19, 2009, to June 11, 2010. On 
average each transcript took 51 minutes to key and code. The time to complete keying and coding 
varied by institution sector, ranging from a 39 minute average for transcripts from both public 2- 
year institutions and private, for-profit 2 years or more institutions, to 77 minutes for transcripts 
from private, nonprofit, 4-year doctorate granting institutions (table 30). 
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Table 30. Average minutes per transcript, by institution control and level: 2009 


Institution control and level 

Number of transcripts 1 

Average 

minutes/transcript 

Total 

8,050 

51.38 

Public 

Less than 2-year 

190 

39.74 

2-year 2 

2,940 

38.77 

4-year non-doctorate granting 

700 

53.52 

4-year doctorate granting 

1,200 

63.96 

Private nonprofit 

2-year or less 

230 

45.00 

4-year non-doctorate granting 

910 

70.63 

4-year doctorate granting 

680 

76.88 

Private for-profit 

Less than 2-year 

630 

45.88 

2 years or more 2 

590 

38.86 


1 Only transcripts completed in less than 4 hours were used in calculation of average minutes/transcript. 

2 Institution offered a 4-year degree during the sample member's enrollment. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 

4.4 Transcript Data Collection Conclusions 

A transcript collection was conducted for BPS:04/ 09 as part of PETS:09. Institution 
contactor staff were trained to facilitate the data collection process, using a transcript control system 
to aid institution representatives in the submission of transcripts. A PETS:09 website was also 
developed to aid institutions in the submission of transcripts, providing instmctions for several 
secure electronic transmission methods, fax, and Federal Express. 

Transcript keying and coding was performed using a specially designed keying and coding 
system that was divided into sections for the entry of data for case information, schools and terms, 
academics, tests, degrees and majors, and courses. A post-data collection debriefing of keyer/ coder 
staff indicated the system was effective for transcript data entry. The PETS coder, created by 
merging 2010 CIP and 2003 CCM, provided a detailed code taxonomy for the coding of courses by 
subject. 

The 5-day keyer/ coder training and ongoing feedback offered through quality circle 
meetings prepared staff to reliably perform keying and coding tasks. All staff passed the proficiency 
test at the conclusion of training, and the results of the keying and course coding interrater reliability 
assessments indicate substantial agreement between keyer/ coders and expert coders. Recoding of 
the random sample of major/ field of study data further supported the reliability of the data. 

Upcoding was performed on all uncoded institutions and additional data elements where 
“other, specify” options were available, such as non-course credit awarded, tests, honors, probations, 
and degree programs. Upcoding added greater detail to data collected. Courses coded with “other” 
codes were reviewed for common subjects, but none were found in great enough numbers to add 
new course codes. 
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Chapter 5. 
Post-Data Collection Data File 
Processing Activities 

The data files for the 2004/ 09 Beginning Postsecondary Students Longitudinal Study 
(BPS:04/ 09) contain student-level data collected from administrative databases, student interviews, 
and transcripts. These data are available to users in two ways. A set of restricted research files fully 
documented through an electronic codebook (ECB) are available to restricted data licensees from 
the National Center for Education Statistics (NCES). Tables and regression analyses can be mn by 
any user through NCES’s online application PowerStats, which also contains variable 
documentation. This chapter describes each file and details the editing and documentation processes 
applied to each. 

5.1 Administrative Record Matching 

In addition to the student interview, data collection for BPS:04/09 included record matching 
to the Central Processing System (CPS), the National Student Loan Data System (NSLDS), and the 
National Student Clearinghouse (NSC) StudentTracker database. This section provides a discussion 
of the observed match rates for these three databases. 

5.1.1 Central Processing System 

The CPS contains data provided to the U.S. Department of Education by students and their 
families when they complete the Free Application for Federal Student Aid (FAFSA). Successful 
record matching to CPS can occur only for sample members who were federal student financial aid 
applicants for the years requested. Matching for BPS:04/ 09 was to CPS data for the 2007-08, 2008- 
09, and 2009-10 financial aid years, using a sample member’s Social Security number (SSN) 
concatenated with the first two letters of the last name as the CPS ID. The percentage of sample 
members who matched to CPS for the 2007-08 academic year was about 28 percent. For 2008-09 
and 2009-10, the rates were approximately 20 percent and 18 percent, respectively. As expected, 
match rates have declined through the years as fewer members of the BPS:04 cohort continue to be 
enrolled in postsecondary education and apply for federal aid. Table 31 provides the CPS matching 
results. 


Table 31. Central Processing System matching results, by academic year: 2007-10 


CPS matching results 



Academic year 



2007- 

-08 

2008- 

-09 

2009- 

-10 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Total 

18,640 

100.0 

18,640 

100.0 

18,640 

100.0 

Matched 

5,120 

27.5 

3,700 

19.9 

3,270 

17.5 

Did not match 

13,530 

72.5 

14,940 

80.1 

15,380 

82.5 


NOTE: Detail may not sum to totals because of rounding. CPS = Central Processing System. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003-04 National Postsecondary 
Student Aid Study (NPSAS:04) and 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). 
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5.1.2 National Student Loan Data System 

The second major database for records matching in BPS:04/09 was the NSLDS. NSLDS 
matching was performed by the NSLDS contractor at the request of the U.S. Department of 
Education, using names, SSNs and dates of birth provided by RTI. Successful matching to NSLDS 
could occur only for sample members who had been awarded federal loans. Pell Grants, National 
Science and Mathematics Access to Retain Talent (SMART) Grants, or Academic Competitiveness 
Grants (ACGs). NSLDS files are historical, so information about a student’s receipt of such loans 
and grants was available not only for the current academic year but also for any prior years. 
Consequendy, match rates reported for the BPS:04/09 sample members do not necessarily reflect 
only the 2008-09 academic year. The federal loan match rate was about 60 percent, and the match 
rate for Pell Grants was about 48 percent. The number of sample members matching to the data 
system for ACGs or SMART Grants was less than 1 percent. This is not surprising, given that these 
grants are relatively new and are only available to undergraduate students. Table 32 summarizes the 
match rates observed for the BPS:04/ 09 sample members. 


Table 32. National Student Load Data System matching results, by loan and grant type: 2009 


NSLDS matching results 

Federal loan 

Pell Grant 

ACG or SMART 
Grant 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Total 

18,640 

100.0 

18,640 

100.0 

18,640 

100.0 

Matched 

11,120 

59.6 

8,930 

47.9 

130 

0.7 

Did not match 

7,520 

40.4 

9,720 

52.1 

18,510 

99.3 


NOTE: Detail may not sum to totals because of rounding. ACG = Academic Competitiveness Grant. 

NSLDS = National Student Load Data System. SMART = Science and Mathematics Access to Retain Talent. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary 
Student Aid Study (NPSAS:04) and 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). 

5.1.3 National Student Clearinghouse 

In addition to the CPS and NSLDS file matching, the BPS:04/09 sample was matched to the 
NSC StudentTracker database, which provides information on postsecondary enrollment, degree, 
and certificate records on behalf of participating postsecondary institutions. In order to perform the 
match, RTI supplied SSNs, names, and dates of birth for sample members to the NSC. Overall, a 
record match for a student’s enrollment at any NSC-participating institution was obtained for about 
58 percent of the BPS:04/ 09 sample. Match results in table 33 are based on enrollment and degree 
records from all participating institutions for the 2006-07 academic year through the 2008-09 
academic year. 


Table 33. National Student Clearinghouse StudentTracker matching results: 2009 


NSC matching results 

Number 

Percent 

Total 

18,640 

100.0 

Matched 

10,820 

58.0 

Did not match 

7,830 

42.0 


NOTE: Detail may not sum to totals because of rounding. NSC = National Student Clearinghouse. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003-04 National Postsecondary 
Student Aid Study (NPSAS:04) and 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09). 
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5.2 BPS:04/09 Main Study Data Files 

The primary analysis file, from which PowerStats was constmcted, contains data for 16,680 
study respondents. A BPS:04/ 09 study respondent was defined as any sample member who was 
determined to be eligible for the study, was still alive at the time of the BPS:04/ 09 data collection, 
and had the requisite valid data from any source to allow construction of his or her enrollment 
history. 

The first data release was adjudicated and approved for public release in December, 2010 
The primary analysis file contains over 1,500 variables, developed from multiple sources. 

Throughout the data collection period, data were processed and examined for quality control (QC) 
purposes. Editing of student data began shortly after the start of web data collection, when 
procedures and programs for this purpose were first developed. Anomalous values were investigated 
and resolved, where appropriate, through the use of data corrections and logical recodes. Interim 
files were delivered to NCES for review throughout the data collection period. 

The restricted-use BPS:04/ 09 ECB contains the following files, each linked by the student’s 
study ID 17 : 

• BPS:04/ 09 analysis file. Contains analytic variables derived from all BPS data sources and 
selected direct student interview variables. 

• BPS:04/ 09 student data file. Contains student interview data collected from 16,680 study 
respondents, which includes 15,160 interview respondents. Topics include enrollment 
history, enrollment characteristics, employment, and background. 

• BPS:04/ 09 institution analysis file. Contains student-level analytic variables derived from 
the BPS:04/ 09 institution data file. 

• BPS:04/ 09 institution data file. Contains institution and degree data obtained from the 
BPS:04/ 09 student interview for all study respondents. It is a student-level file; however, 
a student can have more than one record in the file. There is a separate record for each 
degree obtained from each postsecondary institution that the student attended between 
July 2006 and June 2009 (the maximum number of reported institutions for any one 
respondent was four). 

• BPS:04/ 09 coding data file. Contains major or field of study, industry, and occupation 
strings collected in the BPS student interview and the associated codes. 

• CPS data files. Contains data received from the CPS for the eligible sample members who 
matched to the financial aid application files. 

2003- 04: file contains about 13,780 sample members matched 

2004- 05: file contains about 10,680 sample members matched 

2005- 06: file contains about 8,130 sample members matched 

2006- 07: file contains about 7,160 sample members matched 

2007- 08: file contains about 5,120 sample members matched 


17 The restricted files are available to researchers who have applied for and received authorization from NCES to access the restricted 
ECB. Researchers may obtain authorization by contacting the NCES Data Security Office. 
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2008- 09: file contains about 3,700 sample members matched 

2009- 10: file contains about 3,270 sample members matched 

• NSLDS loan data file. Contains raw loan-level data received from the NSLDS for the 

1 1,120 sample members who were awarded loans from 2003-04 through 2008-09. This 
is a history file with separate records for each transaction in the loan files; therefore, 
there can be multiple records per case spanning several academic years. 

• Veil Grant data file. Contains raw grant-level data received from the NSLDS for the 8,930 
sample members who were awarded Pell Grants from 2003-04 through 2008-09. This is 
a history file with separate records for each transaction in the Pell system; therefore, 
there can be multiple records per case. 

• ACG/ SMAKT data file. Contains raw grant-level data received from the NSLDS for the 
1 30 sample members who were awarded ACGs or SMART Grants from 2003-04 
through 2008-09. This is a history file with separate records for each transaction in the 
database; therefore, there can be multiple records per case. 

• ACT data file. Contains data received from ACT for the 5,940 sample members who 
matched to the 1997-98 through 2002-03 ACT files. 

• 2004 National Postsecondary Student Aid Study (NPSAS:04) file. Contains the base-year data 
included in the NPSAS:04 ECB. 

• BPS:04/ 06 file. Contains all first follow-up data included in the BPS:04/06 ECB. 

• BPS:04/ 09 weights file. Contains all of the analysis weights created for BPS:04/ 09. There is 
a separate record for each study respondent. 18 

5.3 Transcript Data Files 

The data files for the BPS:04/09 transcript component contain the data included on each 
transcript that was entered via the KCS, as well as almost 500 composite variables derived from that 
data. Many of the student-level derived variables are available through PowerStats. 19 The following 
files were produced for the BPS:04/09 Transcript ECB, which contains the 17,000 transcript 
component respondents: 

• Transcript analysis file. Contains over 350 student-level analytic variables derived from 
transcript data, and selected direct transcript variables. 

• Transcript data file. Contains data, such as cumulative transcript totals and high school 
graduation date, from each transcript. Since there is a record for each transcript received, 
this file may contain multiple records per student. 

• Institution data file. Contains institution-level data obtained from the student transcripts 
with a record for each institution that either was sampled and provided transcripts or 
was entered as a transfer institution noted on keyed transcripts. This is a file of 


18 See Chapter 6 for a full description of the BPS:04/09 study weights. 

19 A set of restricted research files fully documented through an electronic codebook (ECB) are available to restricted data licenses 
from the National Center for Education Statistics (NCES). Tables and regression analyses can be run by any user through NCES’s 
online application Powerstats, which also contains variable documentation. Powerstats is available online via the DataLab site at 
http://nces.ed.gov/datalab/index.aspx. 
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institutions only; it does not contain a student ID or transcript ID. Each record includes 
institution control, level, location, credit/ clock hour uses, calendar system, grading 
system, and units required to be designated full-time. This file also contains some 
institution-level derived variables such as institution selectivity and the percentage of 
faculty members who are full-time. 

• Student Schools data file. Contains a record pertaining to a single pairing of student and 
school. That is, for every student there is a record for each school from which we have 
course data, whether from a transcript received from that school or from a transfer 
course listed on a transcript from another school, and therefore there may be multiple 
records per student. Each record contains student ID, school Integrated Postsecondary 
Education Data System (IPEDS) ID, date student first attended institution, transfer 
credits attempted/ accepted at institution, and transfer credits for grade point average. 
This file also contains some student/ school derived variables such as a 72 month 
enrollment string, attendance order, degree attainment indicators, and course cluster 
grade point averages and credits earned. 

• Degree data file. Contains degree-level data with a record for each degree obtained or 
attempted, as listed on any transcript received. Each record includes degree and program 
data, such as type of degree, degree date, and degree honors received. Each record also 
includes majors, minors, concentrations, and their respective 2010 CIP codes. This file 
also contains some degree-level derived variables such as degree order and condensed 
field of study categories. 

• Courses data file. Contains course-level data obtained from the student transcripts with a 
record for each course taken included on any transcript received, as well as transfer 
courses listed. Each record includes course name, course number, grade, credits earned, 
quality points. Postsecondary Education Transcript Study course code, and course 
attributes. This file also contains five course-level derived variables that normalize other 
variables. The normalization process allows for all values of the variable in question to 
be placed on the same scale so that they are comparable across students and institutions. 
This file contains a variable for normalized grade, potential credit, earned credit, and 
quality points. 

• Terms data file. Contains a record pertaining to a single pairing of student and term for all 
institutions. Each record contains the IPEDS ID of the institution, transcript ID, term 
name, start and end dates, and honor s/probation indicator. . This file also contains some 
term-level derived variables such as total earned credits, term grade point average, and 
enrollment status. 

• Transfer data file. Contains a record pertaining to a single transfer event possibility, defined 
as the possible movement of credit when students move from one school to another. . 
Each record contains student ID, origin school IPEDS ID, destination school IPEDS 
ID, credit transfer ratio, and variables that describe the relationship between origin and 
destination institutions with respect to institution and degree program characteristics. 

• W rights file. Contains the analysis weight and replicate weights created for the BPS 
transcript data. There is a separate record for each respondent. 
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5.4 Data Editing 

The BPS:04/ 09 data, including data from the transcript component, were edited using 
procedures developed and implemented for previous studies sponsored by NCES, including the 
base-year study, NPSAS:04, and the first follow-up study, BPS:04/06. Following data collection, the 
information collected in the student instmment and in transcripts was subjected to various QC 
checks and examinations. For example, in the student interview these checks were conducted to 
confirm that the collected data reflected appropriate item routing (skip patterns). Another evaluation 
for both the student interview and transcripts involved examination of all variables with missing data 
and substitution of specific values to indicate the reason for the missing data. For example, an item 
may not have been applicable to particular students, or as in the interview, a respondent may not 
have known the answer to the question or might have skipped the item entirely (table 34). 


Table 34. Description of missing data codes: 2009 


Missing data code 

Description 

-1 

Don’t know 

-3 

Not applicable 

-6 

Value out of range 

-8 

Item was not reached due to an error 

-9 

Data missing 1 


1 Missing data from the abbreviated interview were coded as -9. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary 
Students Longitudinal Study (BPS:04/09). 


Skip-pattern relationships in the interview database were examined by methodically cross- 
tabulating gate items and their associated nested items. In many instances, gate-nest relationships 
spanned multiple levels within the instmment. Items nested within a gate question may themselves 
have been gate items for additional items. Consequently, validating the gate-nest relationships often 
required several iterations and many multiway cross-tabulations to ensure the proper data were 
captured. Gate-nest relationships were also preserved and edited appropriately in the transcript data 
files; however, fewer of these relationships exist in that data. 

The data cleaning and editing process for the BPS:04/09 data files involved a multistage 
process that consisted of the following: 

1. Blank or missing data were replaced with -9 for all variables in the student interview and 
transcript databases. A one-way frequency distribution of every variable was reviewed to 
confirm that no missing or blank values remained. These same one-way frequencies 
revealed any out-of-range, or outlier ; values which were replaced with a -6 value (e.g., 
hourly wages of $0.10, rather than $10.00). Creating SAS formats from expected values 
and the associated value labels also revealed any categorical outliers. Descriptive statistics 
were produced for all continuous variables. All values that were less than zero were 
temporarily recoded to missing, and the minimum, median, maximum, and mean values 
were examined to assess reasonableness of responses; anomalous data patterns were 
investigated and corrected, as necessary. For transcripts, missing data was also replaced 
with a -9 (e.g., if high school graduation date did not appear on the transcript) and one- 
way frequencies were reviewed for any outlier \ alues and also given a -6 value (e.g., credit 
hours of 100 per course, rather than 3). 

2. Legitimate skips were identified with use of instmment source code and flowcharts. 
Gate-nest relationships were defined to replace -9s (data missing, reason unknown ) with -3s 
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(; not applicable), as appropriate. Two-way cross-tabulations between each gate-nest 
combination were evaluated; high numbers of nonreplaced -9 codes were investigated to 
ensure skip-pattern integrity. Nested values were further checked to reveal instances in 
which the legitimate skip code overwrote valid data, which typically occurred if a 
respondent answered a gate question and the appropriate nested items but then reverted 
to change the value of the gate to one that opened on an alternate path of nested items. 
Because responses to the first nested items remained in the database, they required 
editing. For transcripts, gate-nest relationships were limited; however, -3 values were set 
for inapplicable items. For example, if a transcript indicated that the student was still 
working on his or degree, then a -3 value was given to the degree date variable). 

3. Variables were formatted (e.g., dates were formatted as YYYYMM), and time units were 
standardized for items that collected amounts of time in multiple units. In addition, any 
new codes assigned by expert coders reviewing IPEDS, industry, occupation, and major 
codes from the interview (including those strings that could not be coded during the 
interview) were merged back with the interview data files. At this stage, logical recodes 
were performed when the value of missing items could be determined from answers to 
previous questions or preloaded values. For example, if a student was not currently 
repaying education loans, then the monthly payment amount was recoded to $0. For 
transcripts, missing IPEDS, major, and course codes were reviewed and finalized by 
expert coders. 

Concurrently with data cleaning, documentation was developed for both instrument and 
transcript data to detail question text, response options, logical recoding, and the “applies to” text 
for each delivered variable (for documentation information, see the student instmment facsimile in 
appendix D). 

5.5 Data Perturbation 

To protect the confidentiality of information about specific individuals, BPS:04/09 interview 
data were subject to perturbation procedures to minimize disclosure risk. Perturbation procedures, 
which have been reviewed and approved by the NCES Disclosure Review Board, preserve the 
central tendency estimates but may result in slight increases in nonsampling errors. 

In a study like BPS, there are multiple sources of data for some variables (CPS, NSLDS, 
student interview, etc.), and reporting differences can occur in each. Data swapping and other forms 
of perturbation, implemented to protect respondent confidentiality, can lead to inconsistencies as 
well. 

5.6 Statistical Imputations 

All variables from the student interview data and the derived variables in PowerStats with 
missing data were imputed. Imputed data are available in both PowerStats and the restricted derived 
data file. Derived variables obtained from student transcript data and the variables included in the 
remaining restricted files were not imputed. The variables were split into three groups, and a 
consistent imputation methodology was employed for each group. The sequence of variables 
imputed within the three groups depended on the rate and pattern of missing data for the variable 
requiring imputation. The general imputation methodology consisted of two steps. The first step, if 
applicable, was logical or deterministic imputation. If the imputed value could be deduced from the 
logical relationships with other variables, then that information was used to deterministically impute 
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the value for the recipient. The second step was weighted hot-deck imputation. A relatively 
homogenous group of observations was identified, and within this group a random donor’s value 
was selected to impute a value for the recipient. 

Prior to implementing the imputation process, all relationships among the variables were 
checked to ensure that the valid information adhered to the logical relationships. The imputation 
program was designed to impute all missing data as precisely and efficiently as possible using valid 
donor information, such that the process could be completed within a very short timeframe after the 
end of data collection and still maintain the desired quality. The aim was to replace missing data with 
data that were valid in all cases. 

Variables requiring imputation were imputed sequentially. However, some variables that 
were related substantively or had similar levels of missing response were grouped together into 
blocks, and the variables within a block were imputed simultaneously. The order in which variables, 
or blocks of variables, were imputed was primarily based on the level of missing data. The variables 
with lower levels of missing data were imputed before the variables with higher levels of missing 
data. When a variable was selected for imputation based on its level of missing data, three specific 
pieces of information were evaluated. First, logical consistency was checked to make sure that any 
known relationships were maintained throughout the imputation process. Second, the pattern of 
missing data was evaluated to determine whether other variables should be included to create a 
block of variables requiring imputation. Finally, the imputation class variables and sorting variables 
were identified. 

All stochastic imputations used a tree methodology to create imputation classes and the 
weighted sequential hot-deck (WSHD) methodology (Cox 1980; Iannacchione 1982) within 
imputation classes. The imputation classes were formed using nonparametric classification trees 
(Breiman et al. 1984). The nonparametric classification trees formed imputation classes for a 
prediction model based on the observations with valid values for the variable requiring imputation. 
The nonparametric classification tree recursively split the cases into homogenous groups, which 
were used to define the imputation classes. The observations with missing values were assigned their 
imputation class based on the same variables used in the tree splits. 

The WSHD methodology replaced missing data with valid data from a donor record within 
an imputation class. The WSHD methodology also incorporated sorting within imputation classes 
for additional control and used the sample weight of each record in the donor selection process. The 
imputation classes in the application of the WSHD methodology were formed by identifying 
variables related to the variable requiring imputation. Data were sorted within each imputation class 
to increase the chance of obtaining a close match between donor and recipient. Within each 
imputation class, the hot-deck process searched for donors sequentially, starting with the recipient 
and progressing up and down the sorted file to find the set of eligible donors from which a random 
selection of a donor was made. The process was weighted since it incorporated the sample weight of 
each record in the search and selection routine, using the methodology described in Cox (1980). 

Imputation diagnostics consisted of four checks: number of times a donor was used, overall 
imputation checks, imputation checks by class variables, and multivariate consistency checks. The 
check for the number of times a donor was used was to ensure that donors were used a reasonable 
number of times. Using a donor too many times could have indicated that an imputation class had 
too few donors, and that the class needed to be enlarged. The overall imputation checks compared 
the distributions, weighted and unweighted, for each level of the imputed variable before and after 
imputation. Differences greater than 5 percent were flagged and examined to see if changes should 
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be made to the imputation specification. The imputation checks by class variables compared the 
distributions, weighted and unweighted, for each level of the imputed variable in the defined 
imputation classes before and after the imputation. Differences greater than 5 percent were flagged 
for further review. Finally, multivariate consistency checks ensured that relationships between 
variables were maintained and that any special instructions for the imputation were implemented 
properly. 

If any of the four aforementioned diagnostic checks indicated a donor was used too many 
times, substantial deviation from the weighted sums, or any identified inconsistencies, the 
imputation process was revised and rerun. Some results of the imputation process are provided in 
appendix J, which presents the item response and nonresponse rates for each variable subject to 
imputation. Appendix J also contains the means of the continuous variable before and after the 
imputation (the mean of the variables for the cases who responded to the items, and the mean of the 
variables for all cases with either nonmissing or imputed data) and percent distributions of the 
categorical variables before and after imputation. Approximately 17 percent of the variables showed 
statistically significant differences between the pre- and post- imputation means and distributions. 

5.7 Composite and Derived Variable Construction 

Analysts created the main study analytic variables by examining the data available for each 
student from the various data sources, prioritizing the data sources on an item-by-item basis, and 
reconciling discrepancies within and between sources. In some cases, the derived or composite 
variables were created by simple assignment of a value from the available source with the highest 
priority. In other cases, interview items were recoded or otherwise summarized to create a derived 
variable. Similar procedures were used for transcript analytic variables using only data from 
transcripts and institutions providing transcripts. Details about the creation of each variable appear 
in the variable descriptions contained in the ECB and PowerStats. For a listing of the set of analysis 
variables derived for BPS:04/ 09, see appendix K. 
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Weighting and Variance Estimation 

This chapter provides information pertaining to the weighting procedures for the 2004/ 09 
Beginning Postsecondary Students Longitudinal Study (BPS:04/09). The development of statistical 
analysis weights for the BPS:04/ 09 sample, which were derived from the BPS:04/ 06 weights, is 
discussed in section 6.1. Analysis procedures that can be used to produce design-unbiased estimates 
of sampling variances are discussed in section 6.2, including variances computed using Taylor series 
and bootstrap replication techniques. Section 6.2 also describes how the Taylor series strata and 
primary sampling unit (PSU) variables and the bootstrap replicate weights were constmcted. 

Section 6.3 gives weighted and unweighted response rates. Section 6.4 discusses the accuracy of 
BPS:04/ 09 estimates for precision and the potential for nonresponse bias. 

6.1 Analysis Weights 

The weights for analyzing the BPS:04/ 09 data were derived from the BPS:04/ 06 weights 
because the BPS:04/09 respondents are a subset of the BPS:04/06 sample. The BPS:04/06 weights 
were derived from the 2003-04 National Postsecondary Student Aid Study (NPSAS:04) weights 
because the BPS:04/ 06 sample members are a subset of the NPSAS:04 sample. As described in 
chapter 2, all of the BPS:04/ 06 sample members (with the exception of those who were deceased) 
were included in the BPS:04/ 09 data collection. Three weights were developed for analyzing the 
data from the BPS:04/ 09 data collection. One weight was developed for analyzing sample members 
who were considered study respondents for BPS:04/ 09. A second, longitudinal or panel weight, 
was developed for analyzing records of sample members who were considered study respondents 
for the base-year study (NPSAS:04), the first follow-up (BPS:04/06), and the second follow-up 
(BPS:04/09). A third weight was developed for analyzing cases with transcript data. The weights 
were adjusted for nonresponse and were also raked to IPEDS and NPSAS:04 control totals. This 
section describes the steps that were followed in order to develop each weight. 20 

6.1.1 Analysis Weights for BPS:04/09 Study Respondents 

A BPS:04/ 09 study respondent was defined as any sample member who was determined to 
be eligible for the study, was still alive at the time of the BPS:04/ 09 data collection, and had the 
requisite valid data from any source to allow constmction of his or her enrollment history. In 
addition to the BPS:04/ 09 student interview, student-level data for BPS:04/ 09 were collected from 
a variety of administrative sources, including the National Student Loan Data System and the 
National Student Clearinghouse Tracker file, as described in section 2.5. Data from these other 
sources supplemented interview data and allowed enrollment histories and persistence and 
attainment variables to be constmcted for a proportion of interview nonrespondents. Of the 18,640 
students who were BPS-eligible at the conclusion of BPS:04/06, after BPS:04/09 data collection 
there were 110 who were found to be deceased, 15,160 who were interview respondents, 1,520 who 
were not interview respondents but who had enough data from other sources to be classified as 


211 Three analysis weights were also constructed for BPS:96/01: A weight for the analysis of data from BPS:96/01, a weight for the 
analysis of data from NPSAS:96 and BPS:04/06, and a weight for analyses that include data from NPSAS:96, BPS:96/98, and 
BPS:96/01. However, BPS:96/01 did not have a transcript component, and therefore no transcript weight. BPS:96/01 also did not 
use the study respondent definition and did not impute for missing item data; as a result three weights were needed for the analysis 
of interview data compared with the two weights needed for analysis of interview data in BPS:04/09. 
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BPS:04/09 study respondents, and 1,860 who did not have enough data to be considered study 
respondents. Therefore, there were a total of 16,680 study respondents. 

The initial weight for the BPS:04/ 09 cohort was the BPS:04/ 06 analysis weight. An 
adjustment was made for study nonresponse using a model-based constrained logistic weighting 
procedure. The weights were then calibrated to weight sums from BPS:04/06, which had been 
calibrated to IPEDS and NPSAS:04 control totals as described earlier. 21 The procedure 
WTADJUST in SUDAAN (RTI, 2008) was used to implement the nonresponse and calibration 
adjustments. This weighting methodology is described by Folsom and Singh (2000). 

The first adjustment was for study nonresponse. The adjustment model included the 18,540 
cases who were not deceased; the response indicator was set to 1 for the 16,680 study respondents 
and to 0 for the 1,860 cases who were not deceased but were study nonrespondents. Independent 
variables were chosen that were considered to be predictive of response status, and were 
nonmissing for both study respondents and nonrespondents. Variables used in the nonresponse 
adjustment models for BPS and NPSAS surveys were also included. Candidate predictor variables 
included: 


• Institution type, 

• Whether in one of the 12 states with state-representative samples of undergraduates in 
NPSAS:04, 

• OBE (Office of Business Economics) region, 

• Institution enrollment during 2003—04 from the IPEDs file, 

• Age as of December 31, 2003, 

• Gender, 

• Race/ethnicity 

• Citizenship status during the base year, 

• Number of institutions attended, 

• Enrollment pattern during the base year, 

• Whether any dependent children during the base year, 

• Income and dependency status during the base year, 

• Degree plans during the base year, 

• Whether degree was attained through 2006, 

• Persistence and attainment in 2006, 

• Retention and attainment at base year institution through 2006, 

• Parents’ highest education level, 


21 The base weight and the adjustment factors from BPS:04/06 and BPS:04/09 are not included on the BPS data file, 
but can be requested on a separate data set from NCES. 
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• Applied for any aid during the base year, 

• Type of aid package during the base year, 

• Pell Grant recipient, 

• Amount of Pell Grant, 

• Stafford Loan recipient, 

• Amount of Stafford Loan, 

• Federal aid recipient, 

• Institutional aid recipient, 

• State aid recipient, 

• Amount of PLUS (Parents’ Loan for Undergraduate Students) Loan, 

• CPS data available during base year, 

• In field cluster for BPS:04/ 09, 

• NPSAS:04 interview respondent, and 

• BPS:04/ 06 interview respondent. 

Variables included in the nonresponse modeling included all of the candidate predictor 
variables as well as certain important interaction terms. To detect important interactions for the 
nonresponse model, a Chi-squared automatic interaction detection analysis (CHAID) was 
performed on the predictor variables. The CHAID analysis divided the data into segments that 
differed with respect to the response variable. The segmentation process first divided the sample 
into groups based on categories of the most significant predictor of response. It then split each of 
these groups into smaller subgroups based on other predictor variables. It also merged categories of 
a variable that were found to be nonsignificant. CHAID was mn for up to three segments, resulting 
in the identification of two-way and three-way interactions. Variables that made up the CHAID 
interaction terms were whether the student was a BPS:04/ 06 interview respondent, persistence and 
attainment in 2006, retention and attainment through 2006, whether in a field cluster during the 
BPS:04/ 09 data collection, and the state in which the NPSAS:04 institution is located. 

Table 35 lists the predictor variables used in the model to adjust the weight for nonresponse 
and the average weight adjustment factors resulting from these variables. The nonresponse weight 
adjustment factors have the following characteristics: 

• Minimum: 1.00; 

• Median: 1 .02; and 

• Maximum: 2.95. 
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Table 35. Weight adjustment factors for nonresponse for the BPS:04/09 study weights: 2009 

Average 
Weighted weight 

Number of response adjustment 

Model predictor variables respondents rate factor 


Total 

16,680 

89.18 

1.11 

Type of institution 

Public 

Less-than-2-year 

430 

78.98 

1.26 

2-year 

5,570 

87.69 

1.13 

4-year non-doctorate-granting 

1,590 

93.43 

1.06 

4-year doctorate-granting 

2,990 

96.74 

1.03 

Private nonprofit 

Less than 4-year 

440 

85.54 

1.16 

4-year 

3,690 

95.50 

1.04 

Private for-profit 

1,980 

76.27 

1.33 

NPSAS institution control 2003-04 

Public 

10,580 

90.45 

1.10 

Private nonprofit 

4,130 

94.76 

1.05 

Private for-profit 

1,980 

76.27 

1.33 

NPSAS institution level 2003-04 

Less-than-2-year 

1,570 

75.23 

1.35 

2-year 

6,470 

86.68 

1.15 

4-year 

8,640 

94.36 

1.05 

In 12 state NPSAS:04 representative sample 

Yes 

8,340 

89.07 

1.11 

No 

8,350 

89.26 

1.12 

Bureau of Economic Analysis code (Office of Business Economics 
[OBE]) region 

New England (CT ME MA NH Rl VT) 

1,110 

89.67 

1.11 

Mid East (DE DC MD NJ NY PA) 

2,480 

88.15 

1.13 

Great Lakes (IL IN Ml OHWI) 

2,860 

89.45 

1.11 

Plains (IAKS MN MO NE ND SD) 

1,770 

92.09 

1.07 

SE (ALAR FL GA KY LA MS NO SC TN VA WV) 

3,810 

88.66 

1.12 

Southwest (AZ NM OKTX) 

1,760 

87.62 

1.13 

Rocky Mountains (CO ID MT UTWY) 

590 

91.11 

1.11 

Far West (AK CA HI NV OR WA) 

2,080 

89.53 

1.12 

Other jurisdictions (PR) 

250 

92.18 

1.08 

NPSAS:04 institution enrollment size 

1,821 or fewer 

3,860 

82.82 

1.21 

1,822-6,690 

4,210 

89.22 

1.09 

6,691-16,522 

4,250 

90.17 

1.09 

16,523 or more 

4,360 

92.46 

1.07 

Age as of 12/31/03 

15 to 18 

7,920 

93.63 

1.06 

19 

4,320 

91.97 

1.08 

20 to 23 

1,780 

83.26 

1.21 

24 to 29 

1,060 

78.09 

1.30 

30 or older 

1,620 

81.61 

1.25 


See notes at end of table. 
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Table 35. Weight adjustment factors for nonresponse for the BPS:04/09 study weights: 2009 — 
Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Gender 

Male 

6,880 

87.97 

1.13 

Female 

9,810 

90.08 

1.10 

Race/ethnicity 

White 

10,740 

90.78 

1.09 

Black or African American 

2,200 

86.77 

1.15 

Hispanic or Latino 

2,110 

85.33 

1.17 

Asian 

770 

90.68 

1.11 

American Indian or Alaska Native/Native Hawaiian or other Pacific 

Islander/Other/More than one race 

860 

86.36 

1.16 

Citizenship status 2003-04 

US citizen 

15,810 

89.51 

1.11 

Resident alien 

700 

86.07 

1.18 

Foreign or international student 

170 

75.72 

1.31 

Number of institutions attended 2003-04 (NPSAS) 

1 

15,500 

89.35 

1.11 

2 

1,140 

85.93 

1.12 

3 

40 

95.96 

1.03 

Enrollment pattern 2003-04 (NPSAS) 

Enrolled mostly full-time 

13,920 

90.21 

1.10 

Enrolled mostly part-time 

2,380 

85.82 

1.17 

Enrolled full-time & part-time equally 

390 

89.85 

1.11 

Any dependent children, 2003-04 

Yes 

2,350 

79.82 

1.26 

No 

14,340 

91.19 

1.09 

Number of family members in college for independent student, or 
dependent student in 2003-04 

1 

3,460 

81.20 

1.25 

2 

300 

79.46 

1.28 

3 or 4 

50 

94.69 

1.06 

Dependent student 

12,880 

92.36 

1.07 

Total income by dependency 2003-04 

Dependent students 

Less than $10,000 

700 

85.92 

1.16 

$10,000-$1 9,999 

1,040 

88.67 

1.12 

$20,000-$29,999 

1,320 

91.37 

1.08 

$30,000-$39,999 

1,290 

91.16 

1.09 

$40,000-$49,999 

1,150 

91.04 

1.09 

$50,000-$59,999 

1,070 

90.61 

1.10 

$60,000-$69,999 

1,210 

94.34 

1.05 

$70,000-$79,999 

920 

95.11 

1.04 

$80,000-$99,999 

1,560 

95.58 

1.04 

$100,000 or more 

2,630 

94.66 

1.04 


See notes at end of table. 


BPS:04/09 Full-scale Methodology Report 


83 




Chapter 6. Weighting and Variance Estimation 


Table 35. Weight adjustment factors for nonresponse for the BPS:04/09 study weights: 2009 — 
Continued 


Average 
Weighted weight 

Number of response adjustment 

Model predictor variables respondents rate factor 


Independent students 


Less than $10,000 

700 

81.40 

1.23 

$10,000-$1 9,999 

1,370 

79.08 

1.26 

$20,000-$29,999 

640 

79.89 

1.28 

$30,000-$49,999 

600 

83.40 

1.21 

$50,000 or more 

490 

84.34 

1.22 

Degree plans first year 2003-04 

Certificate 

2,250 

77.58 

1.30 

Associate's degree 

5,160 

86.49 

1.14 

Bachelor's degree 

8,020 

95.12 

1.04 

No degree 

1,250 

89.65 

1.11 

Degree types attained through 2006 

Certificate only, associate only, certificate and associate, or 

bachelor only 

2,860 

89.14 

1.13 

Never attained or (Associate and bachelor) 

13,830 

89.19 

1.11 

Highest degree attained anywhere, 2006 

Certificate 

1,380 

82.66 

1.23 

Associate's degree 

1,250 

94.45 

1.05 

Bachelor's degree or never attained 

14,060 

89.32 

1.11 

Persistence and attainment anywhere, 2006 

Attained (still enrolled and not enrolled) 

2,860 

89.15 

1.13 

No degree, still enrolled 

9,380 

96.37 

1.03 

No degree, not enrolled 

4,450 

78.08 

1.27 

Retention and attainment at first institution, 3-year total 2006 

Associate's or bachelor's degree 

1,300 

94.91 

1.05 

Certificate 

1,270 

82.22 

1.24 

No degree, still enrolled 

7,720 

95.40 

1.04 

No degree, transferred 

3,170 

92.36 

1.08 

No degree, left without return 

3,230 

76.56 

1.30 

Parent's highest education level, 2003-04 

Did not complete high school 

1,080 

86.24 

1.17 

High school diploma or equivalent 

4,220 

84.65 

1.18 

Vocational or technical training 

660 

91.36 

1.08 

Less than two years of college 

1,300 

90.74 

1.09 

Associate's degree 

1,230 

88.07 

1.13 

2 or more years of college but no degree 

870 

88.47 

1.12 

Bachelor's degree 

3,580 

93.13 

1.07 

Master's degree or equivalent 

2,310 

93.41 

1.06 

First-professional degree 

430 

96.61 

1.03 

Doctoral degree or equivalent 

640 

93.32 

1.05 

Do not know parent's education level 

370 

80.27 

1.27 


See notes at end of table. 
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Table 35. Weight adjustment factors for nonresponse for the BPS:04/09 study weights: 2009 — 
Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Applied for any aid, 2003-04 

Yes 

14,720 

88.98 

1.11 

No 

1,970 

90.25 

1.10 

Aid package by type of aid, 2003-04 

Grants only 

4,720 

90.18 

1.10 

Loans only 

940 

89.39 

1.16 

Work-study only 

100 

93.99 

1.05 

Other only 

200 

88.63 

1.13 

Grants and loans 

3,430 

85.03 

1.16 

Grants and work-study 

520 

94.90 

1.04 

Grants and other 

250 

88.03 

1.13 

Loans and work-study 

80 

93.80 

1.08 

Loans and other 

330 

94.60 

1.09 

Grants, loans, and work-study 

1,200 

94.96 

1.05 

Grants, loans, and other 

780 

88.25 

1.11 

Grants, work-study, and other 

30 

87.47 

1.17 

(Grants, loans, work-study, and other) or (Work-study and other) 

or (Loans, work-study, and other) 

360 

96.81 

1.03 

No aid received 

3,770 

88.95 

1.12 

Pell Grant recipient 

Yes 

6,060 

85.45 

1.15 

No 

10,630 

91.25 

1.09 

Pell Grant amount received (in dollars) 

None 

10,630 

91.25 

1.09 

$2,000 or less 

2,000 

85.51 

1.17 

$2,001 to $3,700 

2,080 

84.04 

1.16 

$3,701 or more 

1,980 

87.10 

1.13 

Stafford Loan recipient 

Yes 

6,690 

88.05 

1.13 

No 

9,990 

89.78 

1.10 

Stafford Loan amount received (in dollars) 

None 

9,990 

89.78 

1.10 

$2,624 or less 

1,130 

87.78 

1.13 

$2,625 

3,590 

94.32 

1.05 

More than $2,625 

1,980 

79.60 

1.26 

Federal aid recipient 

Yes 

10,140 

88.29 

1.12 

No 

6,540 

90.26 

1.10 

Federal aid amount received (in dollars) 

None 

6,540 

90.26 

1.10 

$2,831 or less 

3,380 

89.20 

1.11 

$2,832 to $6,625 

3,490 

89.33 

1.12 

More than $6,625 

3,270 

86.19 

1.14 


See notes at end of table. 
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Table 35. Weight adjustment factors for nonresponse for the BPS:04/09 study weights: 2009 — 
Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Institutional aid recipient 

Yes 

5,280 

94.07 

1.05 

No 

11,410 

87.52 

1.14 

State aid recipient 

Yes 

4,030 

92.09 

1.08 

No 

12,650 

88.42 

1.12 

PLUS amount received (in dollars) 

None 

15,250 

88.96 

1.12 

$5,825 or less 

470 

90.28 

1.09 

$5,826 to $10,207 

490 

91.65 

1.07 

More than $10,207 

480 

94.74 

1.05 

Data available from Central Processing System (CPS) 2003-04 

Yes 

12,290 

89.09 

1.11 

No 

4,390 

89.38 

1.11 

In field cluster in 2009 

Yes 

940 

63.74 

1.54 

No 

15,740 

91.48 

1.09 

NPSAS:04 interview respondent 

Yes 

16,400 

90.10 

1.11 

No 

290 

73.53 

1.35 

BPS:04/06 interview respondent 

Yes 

13,910 

93.82 

1.06 

No 

2,770 

73.19 

1.35 

Chi-square automatic interaction detection (CHAID) segments 
BPS:04/06 interview nonrespondent 

No degree, still enrolled in 2006 

Base year institution located inAZ, FL, DE, PA, IN, orWV 

190 

73.03 

1.35 

Base year institution located in CO, HI, UT, NE, NV, SC, WA, 
NH, DC, ID, MO, Wl, ME, MT, SD, NC, ND, VT, WY, or 

PR 

220 

98.53 

1.01 

Base year institution located in IA, AK, CT, OH, NJ, IL, GA, 

OR, LA, MN, NM, or Rl 

350 

90.99 

1.08 

Base year institution located in other states 

590 

83.81 

1.19 

No degree, not enrolled anywhere in 2006 

Base year institution located in AL, IA, TX, NH, DE, OK, IN, 

KY, or NC 

260 

71.96 

1.37 

Base year institution located inAK, NY, NU, MA, DC, ID, KS, 
NE, Rl, SC, or WV 

180 

48.27 

2.04 

Base year institution located in CO, CT, HI, OR, ME, SD, ND, 
WA, or WY 

130 

87.77 

1.16 

Base year institution located in other states 

630 

61.28 

1.60 


See notes at end of table. 
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Table 35. Weight adjustment factors for nonresponse for the BPS:04/09 study weights: 2009 — 
Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Attained degree (either still enrolled or not enrolled in 2006) 

Base year institution located inAK, AR, NH, DC, MD, HI, ID, 

NE, MN, MT, SD, NV, NM, ND, Rl, WY, AZ, NJ, TN, or VT 

70 

95.45 

1.02 

Base year institution located in TX, CA, IL, OK, or Ml 

90 

77.31 

1.28 

Base year institution located in other states 

80 

53.69 

1.94 

BPS:04/06 interview respondent 

Attained bachelor's degree at the first institution or no degree 
and still enrolled at the first institution anytime in 2005-06 

Base year institution located inAK, AR, NH, DC, MD, HI, ID, 

NE, MN, MT, SD, NV, NM, ND, Rl, Wl, AZ, NJ, TN, or VT 

1,250 

98.20 

1.02 

Base year institution located in CO, FL, OK, UT, SC, WA, or 

WV 

700 

97.08 

1.03 

Base year institution located in DE, PA, OR, KS, KY, ME, Ml, 
MS, NC, or PR 

1,430 

99.57 

1.00 

Base year institution located in other states 

3,460 

98.36 

1.01 

Attained certificate at first institution or no degree, no transfer, 
and not enrolled at first institution anytime in 2005-06 

Not in BPS:04/09 field cluster 

3,250 

86.81 

1.16 

In BPS:04/09 field cluster 

230 

64.35 

1.58 

No degree, transferred from the first institution by June 2006 

Base year institution located in AK, VA, AR, NH, DC, ID, OR, 

Wl, NE, ME, MT, SD, NV, NM, Rl, VT, WY, CO, CT, NJ, 

PA, IL, MD, HI, MS, ND, WV, or PR 

740 

93.87 

1.06 

Base year institution located in IA, FL, OK, TN, LA, MN, NC, 

OH, SC, WA 

620 

98.93 

1.01 

Base year institution located in other states 

1,180 

96.15 

1.04 

Attained associate's degree at first institution 

Base year institution located in AZ, TX, MA, Wl, IN, MS, CO, 
OH, NH, HI, ID, KS, KY, LA, Ml, MT, SD, NM, NC, ND, Rl, 
SC or WA 

350 

98.91 

1.01 

Base year institution located in VA, CT, NY, NJ, PA, TN, MN, 
or PR 

300 

94.83 

1.05 

Base year institution located in other states 

400 

89.97 

1.10 


NOTE: Detail may not sum to totals because of rounding. Categories of undergraduate enrollment were determined by quartiles. 
Categories of Pell grant amount and Stafford loan amount were determined by tertiles. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09).). 

To ensure population coverage and consistency with NPSAS:04 and BPS:04/06, the 
BPS:04/ 09 study weights were further adjusted to control totals. This adjustment was implemented 
using the SUDAAN WTADJUST procedure. Variables used to define the control totals were the 
same as those used for the calibration-coverage adjustments for NPSAS:04 and BPS:04/06. The 
control totals for the BPS:04/09 weights were obtained using the weighted sums from BPS:04/06 
(using the BPS:04/ 06 analysis weights) for these same variables. 

The following variables were used in defining control totals for the NPSAS:04 study 
weights. NPSAS weight sums were used in defining control totals for BPS:04/ 06, and control totals 
for BPS:04/ 09 were established by summing the BPS:04/ 06 weights for the full cohort (including 
deceased students). 

• Total fall undergraduate enrollment, 
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• Fall undergraduate enrollment by institution type, 

• Non-fall undergraduate enrollment by institution type, 

• Amount of Pell Grants by institution type, and 

• Amount of Stafford Loans by institution type. 

The following variables, derived from the IPEDS 2003 file were also control totals for 
BPS:04/ 06 and the corresponding values were again used for BPS:04/ 09: 

• Number of fall freshmen by institution type, 

• Number of fall full-time freshmen by institution type, 

• Number of fall full-time freshmen receiving federal loans by institution type, 

• Amount of federal loans for fall full-time freshmen, by institution type, 

• Number of fall full-time freshmen receiving federal grants by institution type, and 

• Amount of federal grants for fall full-time freshmen by institution type. 

The control totals include cases who became ineligible for BPS:04/ 09 due to death. Because 
of this, the 110 deceased cases in BPS:04/09 were also included in the calibration adjustment. After 
the adjustment, the deceased cases were dropped from the file; the sum of the final weights 
estimates the number of NPSAS:04 students who were eligible for BPS and were still alive at the 
time of the BPS:04/09 interview. Table 36 shows the variables used for the calibration, the values 
of the control totals, and the average weight adjustment factors for each variable. The last column 
of table 36 shows the sum of the study weights after removing the cases who were deceased at the 
time of the BPS:04/ 09 data collection. Statistics for the weight adjustment factors are the following: 

• Minimum: 0.79, 

• Median: 1.00, and 

• Maximum: 1.37. 

The response adjusted, calibrated study weight is the variable WTA000 on the data file. 
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Table 36. Control totals, weight adjustment factors, and sum of weights for eligible cases for the 
BPS:04/09 study weight raking adjustment: 2009 


Model predictor variables 

Control total 

Study weights - 
average weight 
adjustment factor 

Sum of final 
weights for 
eligible cases 

Total 

3,766,832 

1.00 

3,746,295 

Total fall undergraduate enrollment 

3,216,468 

1.00 

3,199,574 

Fall enrollment, by institution type 

Public 

Less-than-2-year 

43,862 

1.02 

43,700 

2-year 

1,246,603 

1.00 

1,242,966 

4-year non-doctorate-granting 

336,116 

1.00 

333,965 

4-year doctorate-granting 

625,619 

1.00 

623,377 

Private nonprofit 

Less than 4-year 

34,663 

1.01 

34,623 

4-year non-doctorate-granting 

298,980 

1.02 

298,554 

4-year doctorate-granting 

199,524 

0.99 

198,107 

Private for-profit 

Less-than-2-year 

204,002 

0.98 

200,697 

2 years or more 

227,099 

1.02 

223,583 

Non-fall enrollment, by institution type 

Public 

Less-than-2-year 

43,732 

1.01 

43,378 

2-year 

1,513,468 

1.00 

1,505,878 

4-year non-doctorate-granting 

347,399 

0.99 

344,941 

4-year doctorate-granting 

617,794 

1.00 

614,269 

Private nonprofit 

Less than 4-year 

36,724 

1.01 

36,717 

4-year non-doctorate-granting 

319,416 

1.01 

319,078 

4-year doctorate-granting 

195,483 

0.99 

194,066 

Private for-profit 

Less-than-2-year 

209,655 

0.99 

208,671 

2 years or more 

266,008 

1.02 

263,162 

Amount of Pell Grants awarded, by institution type 
(in dollars) 

Public 

Less-than-2-year 

$ 53,948,057 

1.00 

$ 53,793,325 

2-year 

990,010,671 

0.99 

987,176,888 

4-year non-doctorate-granting 

326,046,179 

1.01 

326,046,179 

4-year doctorate-granting 

423,246,907 

1.01 

420,302,200 

Private nonprofit 

Less than 4-year 

82,734,652 

1.01 

82,072,863 

4-year non-doctorate-granting 

273,963,345 

1.05 

273,647,143 

4-year doctorate-granting 

116,529,949 

1.01 

115,766,595 

Private for-profit 

Less-than-2-year 

596,775,900 

1.01 

598,631,871 

2 years or more 

496,981,810 

0.97 

492,562,087 


See notes at end of table. 
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Table 36. Control totals, weight adjustment factors, and sum of weights for eligible cases for the 
BPS:04/09 study weight raking adjustment: 2009 — Continued 


Model predictor variables 

Control total 

Study weights - 
average weight 
adjustment factor 

Sum of final 
weights for 
eligible cases 

Amount of Stafford Loans awarded, by institution 
type (in dollars) 

Public 

Less-than-2-year 

$ 37,729,257 

0.99 

$ 40,072,780 

2-year 

404,153,510 

1.00 

401,473,395 

4-year non-doctorate-granting 

453,062,973 

1.00 

452,192,315 

4-year doctorate-granting 

788,002,590 

1.01 

784,412,732 

Private nonprofit 

Less than 4-year 

68,168,372 

1.01 

70,300,924 

4-year non-doctorate-granting 

611,394,404 

1.02 

610,392,641 

4-year doctorate-granting 

317,667,951 

1.00 

313,355,130 

Private for-profit 

Less-than-2-year 

667,113,041 

1.00 

665,887,187 

2 years or more 

1,121,330,590 

1.01 

1,108,476,038 

Number of fall freshmen, by institution type 

Public 

Less-than-2-year 

43,580 

1.02 

43,363 

2-year 

1,045,056 

1.00 

1,039,308 

4-year non-doctorate-granting 

323,031 

1.00 

320,880 

4-year doctorate-granting 

610,185 

1.00 

607,943 

Private nonprofit 

Less than 4-year 

36,850 

1.01 

36,850 

4-year non-doctorate-granting 

297,938 

1.02 

297,512 

4-year doctorate-granting 

192,614 

0.99 

191,197 

Private for-profit 

Less-than-2-year 

222,068 

0.98 

221,174 

2 years or more 

226,835 

1.03 

224,368 

Fall full-time freshmen enrollment, by institution type 

Public 

Less-than-2-year 

26,072 

1.02 

25,855 

2-year 

631,054 

1.00 

628,624 

4-year non-doctorate-granting 

295,042 

0.99 

292,891 

4-year doctorate-granting 

588,974 

1.00 

587,041 

Private nonprofit 

Less than 4-year 

32,490 

1.01 

32,490 

4-year non-doctorate-granting 

283,273 

1.01 

282,847 

4-year doctorate-granting 

187,564 

1.00 

186,294 

Private for-profit 

Less-than-2-year 

197,579 

0.99 

196,685 

2 years or more 

198,416 

1.04 

196,903 


See notes at end of table. 
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Table 36. Control totals, weight adjustment factors, and sum of weights for eligible cases for the 
BPS:04/09 study weight raking adjustment: 2009 — Continued 


Model predictor variables 

Control total 

Study weights - 
average weight 
adjustment factor 

Sum of final 
weights for 
eligible cases 

Fall full-time freshmen receiving loans, by institution 
type 

Public 

Less-than-2-year 

6,857 

0.99 

6,798 

2-year 

106,739 

1.00 

106,245 

4-year non-doctorate-granting 

131,331 

1.00 

131,000 

4-year doctorate-granting 

259,416 

1.01 

258,939 

Private nonprofit 

Less than 4-year 

14,082 

1.01 

14,082 

4-year non-doctorate-granting 

174,686 

1.02 

174,260 

4-year doctorate-granting 

99,504 

1.00 

98,419 

Private for-profit 

Less-than-2-year 

156,729 

0.99 

156,147 

2 years or more 

185,989 

1.02 

184,539 

Amount of loans for fall full-time freshmen, by 
institution type 

Public 

Less-than-2-year 

$ 31,200,693 

0.99 

$ 30,812,263 

2-year 

288,934,194 

1.00 

287,418,487 

4-year non-doctorate-granting 

457,974,177 

1.00 

456,981,660 

4-year doctorate-granting 

960,248,387 

1.01 

958,683,761 

Private nonprofit 

Less than 4-year 

63,879,653 

1.01 

63,879,653 

4-year non-doctorate-granting 

782,232,048 

1.02 

780,073,727 

4-year doctorate-granting 

506,223,469 

1.00 

500,662,774 

Private for-profit 

Less-than-2-year 

671,241,887 

0.99 

670,258,172 

2 years or more 

1,051,784,691 

1.02 

1,046,332,274 

Fall full-time freshmen receiving federal grants, by 
institution type 

Public 

Less-than-2-year 

18,751 

1.00 

18,573 

2-year 

245,138 

1.00 

244,510 

4-year non-doctorate-granting 

104,860 

1.02 

104,860 

4-year doctorate-granting 

152,951 

1.01 

152,073 

Private nonprofit 

Less than 4-year 

24,491 

1.00 

24,491 

4-year non-doctorate-granting 

101,212 

1.05 

100,945 

4-year doctorate-granting 

42,725 

1.01 

42,537 

Private for-profit 

Less-than-2-year 

193,758 

1.01 

192,864 

2 years or more 

157,166 

1.00 

156,518 


See notes at end of table. 
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Table 36. Control totals, weight adjustment factors, and sum of weights for eligible cases for the 
BPS:04/09 study weight raking adjustment: 2009 — Continued 


Model predictor variables 

Control total 

Study weights - 
average weight 
adjustment factor 

Sum of final 
weights for 
eligible cases 

Amount of federal grants for fall full-time freshmen, 
by institution type 

Public 

Less-than-2-year 

$45,841,205 

1.00 

$ 45,434,148 

2 -year 

669,724,132 

1.00 

667,666,709 

4-year non-doctorate-granting 

314 , 444,744 

1.02 

314 , 444,744 

4-year doctorate-granting 

462,635,038 

1.01 

459,752,080 

Private nonprofit 

Less than 4-year 

79,273,524 

1.00 

79,273,524 

4-year non-doctorate-granting 

316,011,424 

1.05 

315,420,406 

4-year doctorate-granting 

157,828,815 

1.01 

157,065,462 

Private for-profit 

Less-than-2-year 

570,341,901 

1.01 

570,173,299 

2 years or more 

433,973,817 

1.00 

432,959,930 


NOTE: Deceased cases are included in the “Control total” column but are not included in the “Sum of final weights for eligible 
cases” column and as a result the two columns are not identical. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

Table 37 summarizes the weight distributions and the variance inflation due to unequal 
weighting by type of institution. The median student study weight ranges from 67 for students 
whose base year institution was public less-than-2-year to 220 for students whose base year 
institution was private for-profit 2 years or more. The mean student study weight ranges from 97 
for students whose base year institution was private nonprofit less than 4-year to 309 for students 
whose base year institution was private for-profit 2 years or more. The unequal weighting effect 
overall is 2.09, and ranges from 1.34 for students whose base year institution was public 4-year 
doctorate -granting to 3.06 for students whose base year institution was private nonprofit less than 
4-year. 
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Table 37. Weight distribution and unequal weighting effects for the BPS:04/09 study weight 
(WTA000), by type of institution: 2009 


Type of institution 

Minimum 

First 

quartile 

Median 

Third 

quartile 

Maximum 

Mean 

Unequal 

weighting 

effect 

Total 

0.90 

95.19 

166.90 

270.94 

5053.00 

224.54 

2.09 

Institutional sector 

Public 

Less-than-2-year 

6.73 

36.38 

67.12 

131.24 

934.42 

114.82 

2.50 

2-year 

2.63 

120.01 

202.26 

336.16 

5053.00 

290.28 

2.10 

4-year non-doctorate-granting 

0.90 

110.53 

210.52 

303.96 

1962.17 

235.24 

1.60 

4-year doctorate-granting 

3.25 

127.81 

216.88 

263.60 

1414.91 

208.51 

1.34 

Private nonprofit 

Less than 4-year 

1.66 

39.36 

69.26 

111.96 

1824.08 

96.50 

3.06 

4-year non-doctorate-granting 

1.04 

72.15 

132.94 

192.32 

1762.41 

155.12 

1.83 

4-year doctorate-granting 

1.27 

79.63 

115.88 

146.90 

2164.40 

127.18 

1.62 

Private for-profit 

Less-than-2-year 

1.34 

24.46 

146.13 

309.46 

1781.71 

214.77 

2.38 

2 years or more 

2.85 

128.50 

219.65 

385.73 

3095.47 

309.05 

2.01 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


To assess the overall predictive ability of the nonresponse model, a Receiver Operating 
Characteristic (ROC) curve was used (Hanley and McNeil 1982). The ROC provides a measure of 
how well the model correctly classified individuals of known response type. For a more detailed 
example of the use of the ROC curve in nonresponse modeling, see Iannacchione (2003). The 
ROC curve was developed by calculating, for any specified probability, c, two proportions: 

• the proportion of respondents with a predicted probability of response greater than c, 
and 

• the proportion of nonrespondents with a predicted probability of response greater than 
c. 

The predicted probability of response for each student was the predicted response 
probability from the weight adjustment model. The plot of the first probability against the second, 
for c ranging from 0 to 1, resulted in the ROC curve shown in figure 22. The area under the curve 
measures the probability that a randomly chosen pair of observations — one respondent and one 
nonrespondent — will be correctly ranked. The probability of a correct pairwise ranking is the same 
quantity that is estimated by the nonparametric Wilcoxon statistic. The null hypothesis associated 
with the Wilcoxon statistic is that the variable is not a useful discriminator between the respondent 
and nonrespondent populations. This corresponds to the null hypothesis that the predicted 
response probability of a respondent is just as likely to be smaller than the predicted response 
probability of a nonrespondent as it is to be greater. Thus, if the null hypothesis is tme, the ROC 
curve will be a diagonal line that reflects the equally likely chance of making a correct or incorrect 
decision, and the area under the curve will be 0.5. If the null hypothesis is not tme, the ROC curve 
will rise above the diagonal and the area under the curve will be significantly greater than 0.5. Figure 
22 shows that the area under the ROC curve is 0.74 such that 74 percent of the time (or more than 
7 of 10 pairings), the predicted probabilities give the correct classification. The ROC area of 0.74 
equals the value of the Wilcoxon test statistic, failing to support the null hypothesis of no predictive 
ability (p < 0.05). This level of discrimination implies that the variables used in the model are highly 
informative, but not definite predictors of a sample student’s overall response propensity. 
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Figure 22. Receiver operating characteristics (ROC) curve for BPS:04/09 study response 
propensity: 2009 



P(response | respondent) 


Observed ROC— area under the curve = 0.73766 

50%predictive ROC 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

6.1.2 Analysis Weights for Study Respondents to NPSAS:04, BPS:04/06, and 
BPS:04/09 (Longitudinal or Panel weight) 

A weight was also constructed for analyzing the cases who were study respondents to all 
three of the base-year study (NPSAS:04), the first follow-up study (BPS:04/06), and the second 
follow-up study (BPS:04/09). This weight is referred to as the “longitudinal” or “panel” weight. As 
described in the NPSAS:04 Methodology Report (Cominole et al.), NPSAS:04 study respondents 
were required to have key data items from either the NPSAS:04 student interview or other external 
data files. The requirements for being a study respondent for BPS:04/ 09 are described in section 
2.5 of this report. BPS:04/06 did not utilize a study respondent definition, but study respondents to 
BPS:04/ 06 were defined for this weight as students who had either BPS:04/ 06 interview data or 
enrollment data from another source; this is similar to the study respondent definition used for 
BPS:04/09. Of the 18,640 students who were BPS-eligible at the conclusion of BPS:04/06, after 
BPS:04/09 data collection there were 110 who were found to be deceased, 16,120 who met the 
definition of a “panel respondent,” and 2,420 who were considered nonrespondents for the panel 
analysis weight. 

As with the study weight described in section 6.1.1, the initial weight was the BPS:04/06 
analysis weight. An adjustment was made for panel nonresponse using a model-based constrained 
logistic weighting procedure. The weights were then calibrated to the sums of the BPS:04/ 09 study 
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weights for eligible cases. The procedure WTADJUST in SUDAAN was used to implement the 
nonresponse and calibration adjustments. 

The first adjustment was for panel nonresponse. The adjustment model included the 18,540 
cases who were not deceased; the response indicator was set to 1 for the 16,120 panel respondents 
and set to 0 for the 2,420 cases who were nondeceased panel nonrespondents. Predictor variables 
were chosen if considered to be predictive of response status, and were nonmissing for both study 
respondents and nonrespondents. Variables used in the nonresponse adjustment models for both 
NPSAS and BPS were also included. Candidate predictor variables included the same set of 
variables that was used for the study weight (see section 6.1.1). 

Variables included in the nonresponse modeling included all of the candidate predictor 
variables as well as certain important interaction terms identified using CHAID. CHAID was mn 
for up to three segments, resulting in the identification of two-way and three-way interactions. Six 
variables made up the CHAID interaction terms: whether the student was a BPS:04/ 06 interview 
respondent, retention and attainment through 2006, type of aid in the base year, total federal aid in 
the base year, base year institution type, and the state in which the NPSAS:04 institution is located. 

Table 38 shows the predictor variables used in the model to adjust the weight and the 
average weight adjustment factors resulting from these variables. The nonresponse adjustment 
factors have the following characteristics: 

• Minimum: 1.00; 

• Median: 1 .02; and 

• Maximum: 3.00 
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Table 38. Weight adjustment factors for nonresponse for the BPS:04/09 longitudinal (panel) 
weights: 2009 

Average 

Weighted weight 

Number of response adjustment 

Model predictor variables respondents rate factor 


Total 

16,120 

85.69 

1.15 

Type of institution 

Public 

Less-than-2-year 

380 

71.29 

1.39 

2-year 

5,360 

83.63 

1.18 

4-year non-doctorate-granting 

1,560 

91.72 

1.08 

4-year doctorate-granting 

2,950 

94.97 

1.04 

Private nonprofit 

Less than 4-year 

420 

80.26 

1.25 

4-year 

3,640 

93.67 

1.05 

Private for-profit 

1,810 

69.99 

1.47 

NPSAS institution control 2003-04 

Public 

10,250 

87.19 

1.13 

Private nonprofit 

4,060 

92.67 

1.07 

Private for-profit 

1,810 

69.99 

1.47 

NPSAS institution level 2003-04 

Less-than-2-year 

1,390 

65.90 

1.53 

2-year 

6,220 

82.73 

1.19 

4-year 

8,510 

92.40 

1.06 

In 12 state NPSAS:04 representative sample 

Yes 

8,050 

85.11 

1.15 

No 

8,070 

86.12 

1.16 

Bureau of Economic Analysis code (Office of Business Economics 
[OBE]) region 

New England (CT ME MA NH Rl VT) 

1,080 

87.35 

1.13 

Mid East (DE DC MD NJ NY PA) 

2,390 

85.42 

1.15 

Great Lakes (IL IN Ml OHWI) 

2,790 

87.42 

1.13 

Plains (IAKS MN MO NE ND SD) 

1,710 

88.70 

1.11 

SE (AL AR FL GA KY LA MS NO SC TN VA WV) 

3,690 

85.62 

1.16 

Southwest (AZ NM OKTX) 

1,650 

80.34 

1.24 

Rocky Mountains (CO ID MT UT WY) 

580 

89.64 

1.14 

Far West (AK CA HI NV OR WA) 

2,000 

85.16 

1.18 

Other jurisdictions (PR) 

240 

88.23 

1.14 

Institution enrollment size 

1,821 or fewer 

3,630 

78.09 

1.29 

1,822 to 6,690 

4,090 

85.84 

1.12 

6,691 to 16,522 

4,130 

87.29 

1.13 

16,523 or more 

4,270 

89.15 

1.10 

Age as of 12/31/03 

15-18 

7,750 

91.15 

1.08 

19 

4,210 

89.13 

1.11 

20-23 

1,660 

77.50 

1.31 

24-29 

990 

73.55 

1.36 

30 or older 

1,510 

76.39 

1.34 


See notes at end of table. 
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Table 38. Weight adjustment factors for nonresponse for the BPS:04/09 longitudinal (panel) 
weights: 2009 — Continued 

Average 

Weighted weight 

Number of response adjustment 

Model predictor variables respondents rate factor 

Gender 


Male 

6,630 

84.31 

1.17 

Female 

9,500 

86.71 

1.14 

Race/ethnicity 

White 

10,450 

88.12 

1.12 

Black or African American 

2,100 

82.24 

1.21 

Hispanic or Latino 

1,990 

79.71 

1.25 

Asian 

750 

86.81 

1.17 

American Indian or Alaska Native/Native, Hawaiian or other 

Pacific Islander, Other, More than one race 

830 

82.23 

1.23 

Citizenship status 2003-04 (NPSAS) 

US citizen 

15,310 

86.07 

1.15 

Resident alien 

670 

83.32 

1.22 

Foreign or international student 

140 

65.14 

1.59 

Number of institutions attended 2003-04 (NPSAS) 

1 

14,970 

85.77 

1.16 

2 

1,110 

84.00 

1.13 

3 

40 

95.96 

1.03 

Enrollment pattern 2003-04 (NPSAS) 

Enrolled mostly full-time 

13,490 

87.25 

1.14 

Enrolled mostly part-time 

2,260 

80.87 

1.24 

Enrolled full-time & part-time equally 

370 

84.37 

1.20 

Any dependent children 2003-04 (NPSAS) 

Yes 

2,210 

74.59 

1.36 

No 

13,910 

88.08 

1.12 

Number of family members in college for independent student, or 
dependent student in 2003-04 

1 

3,250 

76.12 

1.34 

2 

280 

72.91 

1.35 

3 or 4 

40 

86.81 

1.19 

Dependent student 

12,550 

89.57 

1.10 

Total income by dependency (categorical) 2003-04 

Dependent 

Less than $10,000 

670 

82.93 

1.20 

$10,000-$1 9,999 

1,000 

84.40 

1.19 

$20,000-$29,999 

1,280 

88.07 

1.11 

$30,000-$39,999 

1,260 

88.32 

1.11 

$40,000-$49,999 

1,120 

87.61 

1.14 

$50,000-$59,999 

1,040 

87.92 

1.13 

$60,000-$69,999 

1,180 

91.88 

1.07 

$70,000-$79,999 

890 

92.14 

1.07 

$80,000-$99,999 

1,530 

94.24 

1.05 

$100,000 or more 

2,570 

92.21 

1.07 


See notes at end of table. 
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Table 38. Weight adjustment factors for nonresponse for the BPS:04/09 longitudinal (panel) 
weights: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Independent 

Less than $10,000 

670 

76.59 

1.30 

$10,000-$1 9,999 

1,270 

72.37 

1.39 

$20,000-$29,999 

610 

76.45 

1.32 

$30,000-$49,999 

570 

79.35 

1.28 

$50,000 or more 

460 

78.56 

1.34 

Degree plans first year, 2003-04 

Certificate 

2,040 

70.37 

1.44 

Associate's degree 

4,990 

82.95 

1.19 

Bachelor's degree 

7,900 

93.08 

1.06 

No degree 

1,200 

85.14 

1.17 

Degree types attained through 2006 

Attained 

2,740 

85.93 

1.18 

Never attained 

13,380 

85.65 

1.15 

Highest degree attained anywhere, 2006 

Certificate 

1,270 

76.38 

1.32 

Associate's or bachelor's degree 

1,240 

94.01 

1.06 

Never attained 

13,610 

85.82 

1.15 

Persistence and attainment anywhere, 2006 

Attained, still enrolled 

1,170 

90.90 

1.08 

Attained, not enrolled 

1,570 

81.84 

1.25 

No degree, still enrolled 

9,100 

92.95 

1.06 

No degree, not enrolled 

4,280 

74.35 

1.34 

Retention and attainment at first institution, 3-year total 2006 

Associate's or bachelor's degree 

1,300 

94.49 

1.06 

Certificate 

1,150 

75.62 

1.33 

No degree, still enrolled 

7,490 

92.10 

1.07 

No degree, transferred or left without return 

6,190 

79.27 

1.24 

Parent's highest education level, 2003-04 

Did not complete high school 

1,010 

79.88 

1.28 

High school diploma or equivalent 

4,020 

80.80 

1.23 

Vocational or technical training 

640 

86.10 

1.16 

Less than two years of college 

1,260 

86.84 

1.14 

Associate's degree 

1,200 

85.46 

1.16 

2 or more years of college but no degree 

840 

84.65 

1.17 

Bachelor's degree 

3,500 

90.63 

1.09 

Master's degree or equivalent 

2,260 

90.74 

1.08 

First-professional degree 

420 

95.22 

1.05 

Doctoral degree or equivalent 

630 

90.72 

1.06 

Do not know parent's education level 

350 

76.88 

1.32 

Applied for any aid, 2003-04 

Yes 

14,240 

85.50 

1.16 

No 

1,880 

86.71 

1.15 


See notes at end of table. 
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Table 38. Weight adjustment factors for nonresponse for the BPS:04/09 longitudinal (panel) 
weights: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Aid package by type of aid, 2003-04 

Grants only 

4,560 

86.42 

1.15 

Loans only 

900 

86.16 

1.20 

Work-study only 

90 

88.30 

1.10 

Other only 

180 

83.00 

1.21 

Grants and loans 

3,350 

81.95 

1.20 

Grants and work-study 

500 

89.50 

1.10 

Grants and other 

240 

84.77 

1.17 

Loans and work-study 

80 

89.18 

1.13 

Loans and other 

320 

93.90 

1.11 

Work-study and other 

10 

74.40 

1.43 

Grants, loans, and work-study 

1,190 

93.99 

1.04 

Grants, loans, and other 

770 

86.46 

1.12 

Grants, work-study, and other 

30 

84.26 

1.16 

Loans, work-study, and other 

30 

99.96 

1.00 

Grants, loans, work-study, and other 

320 

95.99 

1.04 

No aid received 

3,570 

84.79 

1.18 

Pell Grant recipient 

Yes 

5,870 

81.93 

1.20 

No 

10,260 

87.78 

1.13 

Pell Grant amount received (in dollars) 

None 

10,260 

87.78 

1.13 

$2,000 or less 

1,940 

81.63 

1.22 

2,001 to $3,700 

2,020 

80.92 

1.21 

More than $3,700 

1,910 

83.55 

1.18 

Stafford Loan recipient 

Yes 

6,550 

85.77 

1.15 

No 

9,570 

85.65 

1.16 

Stafford Loan amount received (in dollars) 

None 

9,570 

85.65 

1.16 

$2,625 or less 

1,090 

84.08 

1.16 

$2,625 

3,560 

93.80 

1.06 

More than $2,625 

1,900 

75.70 

1.32 

Federal aid recipient 

Yes 

9,870 

85.35 

1.16 

No 

6,250 

86.10 

1.15 

Federal aid amount received (in dollars) 

None 

6,250 

86.10 

1.15 

$2,831 or less 

3,270 

85.66 

1.15 

$2,832 to $6,625 

3,410 

86.46 

1.16 

More than $6,625 

3,200 

83.88 

1.16 

Institutional aid recipient 

Yes 

5,190 

91.48 

1.07 

No 

10,930 

83.73 

1.19 


See notes at end of table. 
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Table 38. Weight adjustment factors for nonresponse for the BPS:04/09 longitudinal (panel) 
weights: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

State aid recipient 

Yes 

3,940 

89.31 

1.11 

No 

12,180 

84.74 

1.17 

PLUS amount received (in dollars) 

None 

14,710 

85.27 

1.16 

$5,825 or less 

460 

89.38 

1.10 

$5,826 to $10,207 

480 

91.00 

1.08 

More than $10,207 

470 

93.92 

1.06 

Data available from Central Processing System (CPS) 2003-04 

Yes 

11,980 

86.39 

1.14 

No 

4,140 

84.20 

1.19 

In field cluster 

Yes 

820 

55.48 

1.74 

No 

15,300 

88.42 

1.12 

NPSAS:04 interview respondent 

Yes 

15,880 

86.91 

1.15 

No 

250 

64.98 

1.52 

BPS:04/06 interview respondent 

Yes 

13,900 

93.70 

1.07 

No 

2,220 

58.11 

1.68 

Chi-square automatic interaction detection (CHAID) segments 

No degree, no transfer, and not enrolled at first institution anytime 
in 2005-06 

Base year institution located in AR, OH, KS, WA, PR, AL, IA, 

CT, DE, IN, KY, OR, or ND and BPS:04/06 interview 
respondent 

590 

97.05 

1.02 

Base year institution located in AR, OH, KS, WA, PR, AL, IA, 

CT, DE, IN, KY, OR, or ND and BPS:04/06 interview 
nonrespondent 

140 

80.33 

1.20 

Base year institution located in MO, Wl, NC, or SC and no 
federal aid 

80 

95.46 

1.02 

Base year institution located in MO, Wl, NC, or SC and 
received federal aid 

140 

97.53 

1.03 

Base year institution located in MD, WV, A Z, OK, IL, MS, NH, 

AK, or AR and BPS:04/06 interview respondent 

270 

91.50 

1.09 

Base year institution located in MD, WV, A Z, OK, IL, MS, NH, 

AK, or AR and BPS:04/06 interview nonrespondent 

70 

59.83 

1.58 

Base year institution located in VA, LA, NE, ME, ID, SD, WY, 

DC, MT, or VT 

210 

99.49 

1.00 

Base year institution located in other states and BPS:04/06 
interview respondent 

1,300 

96.03 

1.04 

Base year institution located in other states and BPS:04/06 
interview nonrespondent 

280 

66.61 

1.50 


See notes at end of table. 
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Table 38. Weight adjustment factors for nonresponse for the BPS:04/09 longitudinal (panel) 
weights: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

No degree, transferred from first institution by June 2006 

Base year institution located in AR, AZ, IA, CT, IN, KY, MS, NE, 
WY, UT, NH, AK, VT, FL, TN, OH, VA, WA, PR, or Rl and 
BPS:04/06 interview respondent 

870 

84.56 

1.20 

Base year institution located in ID, WV, NJ, PA, OK, AL, DE, IL, 
Wl, LA, MN, NV, NO, OR, SC, DC, MT, or ME and 

BPS:04/06 interview respondent 

650 

88.89 

1.13 

Base year institution located in CO, KS, or MAand BPS:04/06 
interview respondent 

90 

70.22 

1.45 

Base year institution located in other states and BPS:04/06 
interview respondent 

790 

83.27 

1.22 

Base year institution located in AR, WV, NJ, CA, OK, NY, NE, 

NM, DC, MT and BPS:04/06 interview nonrespondent 

120 

33.53 

2.18 

Base year institution located in TN, CO, IL, Wl, IN, NV, TX, UT, 
or AK and BPS:04/06 interview nonrespondent 

210 

58.90 

2.09 

Base year institution located in MO, MD, PR, AL, DE, or SC 
and BPS:04/06 interview nonrespondent 

30 

20.83 

1.76 

Base year institution located in WA, CT, KY, SD, NO, OR, WY, 

HI, or ME and BPS:04/06 interview nonrespondent 

90 

82.89 

1.73 

Base year institution located in other states and BPS:04/06 
interview nonrespondent 

240 

47.28 

2.13 

No degree, still enrolled at first institution anytime in 2005-06 

Base year institution located in AR, IA, Wl, MS, NE, NV, NO, 

ME, ND 

870 

98.35 

1.01 

Base year institution located in FL, AZ, OK, or NM and 

BPS:04/06 interview respondent 

400 

96.95 

1.03 

Base year institution located in FL, AZ, OK, or NM and 

BPS:04/06 interview nonrespondent 

70 

47.01 

2.09 

Base year institution located in other states and aid package 
was grants, loans, and work study 

370 

98.55 

1.01 

Base year institution located in other states and aid package is 
loans only, work study only, grants and loans, or loans and 
work study 

1,420 

90.33 

1.09 

Base year institution located in other states and other types of 
aid packages 

1,370 

93.61 

1.06 

Base year institution located in WY, NH, MT, AK, VT, OH, CO, 

VA, PA, IN, KY, LA, OR, SC, or HI 

1,590 

95.16 

1.05 

Base year institution located in WV, NJ, WA, AL, DE, GA, SD, 

TX, or DC and type of institution is public less-than-2-year, 
private-for-profit less-than-2-year, or private-for-profit 2 
years or more 

30 

37.56 

1.74 

Base year institution located in WV, NJ, WA, AL, DE, GA, SD, 

TX, or DC and type of institution is public 2-year 

430 

85.11 

1.23 

Base year institution located in WV, NJ, WA, AL, DE, GA, SD, 

TX, or DC and type of institution is public 4-year non- 
doctorate-granting, public 4-year doctorate granting, 
private nonprofit less than 4-year, private-nonprofit 4-year 
non-doctorate-granting, or private-nonprofit 4-year 
doctorate granting. 

930 

96.92 

1.02 


See notes at end of table. 
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Table 38. Weight adjustment factors for nonresponse for the BPS:04/09 longitudinal (panel) 
weights: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Attained certificate at first institution 

Base year institution located in AR, KS, VA, NJ, IA, GA, Wl, 

KY, LA, MN, NV, NM and BPS:04/06 interview respondent 

230 

92.28 

1.10 

Base year institution located in ID, WV, PR, DE, NE, NC, SC, 

WY, NH, HI, Rl, MD, AZ, CT, MS and BPS:04/06 interview 
respondent 

110 

76.67 

1.52 

Base year institution located in PA, MA, UT, DC, ME and 
BPS:04/06 interview respondent 

80 

74.20 

1.47 

Base year institution is located in other states and BPS:04/06 
interview respondent 

80 

34.93 

2.74 

BPS:04/06 interview nonrespondent 

660 

85.51 

1.20 

Attained bachelor's degree at first institution, or attained 

associate’s degree at first institution and base year institution 
located in FL, KS, MO, WV, CA, AL, IA, DE, IL, NE, NV, OR, 
WY, OH, ID, CO, WA, KY, LA, Ml, SD, NM, NC, SC, NH, HI, 

MT, Rl, or ND 

720 

94.08 

1.06 

Attained associate's degree at first institution and base year 
institution is located in other states and age 23 or younger in 
2003 

490 

95.64 

1.04 

Attained associate's degree at first institution and base year 
institution is located in other states and age 24 or older in 

2003 

90 

92.27 

1.10 


NOTE: Detail may not sum to totals because of rounding. Categories of undergraduate enrollment were determined by quartiles. 
Categories of Pell grant amount and Stafford loan amount were determined by tertiles. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

To ensure population coverage and consistency with the BPS04/09 study weight, and the 
BPS:04/ 06 and NPSAS:04 weights, the BPS:04/ 09 panel weight was adjusted to control totals 
determined by the BPS:04/ 09 study weight sums. Deceased cases were not included in either the 
control totals or the set of cases in the adjustment. This adjustment was also implemented using the 
SUDAAN WT ADJUST procedure. The control totals for the BPS:04/ 09 panel weights were 
obtained by summing the BPS:04/09 study weights (WTA000). The control variables were the 
same enrollment and aid variables that were used for the BPS:04/ 09 study weight and the 
BPS:04/06 analysis weight. Table 39 presents the variables used for the calibration, the values of 
the control totals, and the average weight adjustment factors for each variable. Statistics for the 
nonresponse weight adjustment factors are the following: 

• Minimum: 0.69; 

• Median: 1 .00; and 

• Maximum: 1.46. 

The response-adjusted, calibrated panel weight is the variable WTB000 on the data file. 
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Table 39. Control totals and sums of weights for eligible cases for the BPS:04/09 panel weight 
raking adjustment: 2009 


Model predictor variables 

Control total 

Panel weights — 
average weight 
adjustment factor 

Total 

3,746,295 

1.00 

Total fall undergraduate enrollment 

3,199,574 

1.00 

Fall enrollment, by institution type 

Public 

Less-than-2-year 

43,700 

1.04 

2-year 

1,242,966 

1.00 

4-year non-doctorate-granting 

333,965 

0.99 

4-year doctorate-granting 

623,377 

1.00 

Private nonprofit 

Less than 4-year 

34,623 

1.03 

4-year non-doctorate-granting 

298,554 

1.02 

4-year doctorate-granting 

198,107 

0.98 

Private for-profit 

Less-than-2-year 

200,697 

1.01 

2 years or more 

223,583 

1.05 

Non-fall enrollment, by institution type 

Public 

Less-than-2-year 

43,378 

1.03 

2-year 

1,505,878 

1.00 

4-year non-doctorate-granting 

344,941 

0.99 

4-year doctorate-granting 

614,269 

1.00 

Private nonprofit 

Less than 4-year 

36,717 

1.02 

4-year non-doctorate-granting 

319,078 

1.02 

4-year doctorate-granting 

194,066 

0.98 

Private for-profit 

Less-than-2-year 

208,671 

1.00 

2 years or more 

263,162 

1.01 


Amount of Pell Grants awarded, by institution type (in dollars) 
Public 


Less-than-2-year 

$ 53,793,325 

1.03 

2-year 

987,176,888 

0.99 

4-year non-doctorate-granting 

326,046,179 

1.01 

4-year doctorate-granting 

420,302,200 

1.01 

Private nonprofit 

Less than 4-year 

82,072,863 

1.01 

4-year non-doctorate-granting 

273,647,143 

1.05 

4-year doctorate-granting 

115,766,595 

1.00 

Private for-profit 

Less-than-2-year 

598,631,871 

1.01 

2 years or more 

492,562,087 

0.96 


See notes at end of table. 
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Table 39. Control totals and sums of weights for eligible cases for the BPS:04/09 panel weight 
raking adjustment: 2009 — Continued 


Model predictor variables 

Control total 

Panel weights — 
average weight 
adjustment factor 

Amount of Stafford Loans awarded, by institution type (in dollars) 

Public 

Less-than-2-year 

$ 40,072,780 

1.05 

2-year 

401,473,395 

0.99 

4-year non-doctorate-granting 

452,192,315 

1.01 

4-year doctorate-granting 

784,412,732 

1.01 

Private nonprofit 

Less than 4-year 

70,300,924 

1.05 

4-year non-doctorate-granting 

610,392,641 

1.03 

4-year doctorate-granting 

313,355,130 

0.98 

Private for-profit 

Less-than-2-year 

665,887,187 

1.00 

2 years or more 

1,108,476,038 

0.99 

Number of fall freshmen, by institution type 

Public 

Less-than-2-year 

43,363 

1.04 

2-year 

1,039,308 

0.99 

4-year non-doctorate-granting 

320,880 

1.00 

4-year doctorate-granting 

607,943 

1.00 

Private nonprofit 

Less than 4-year 

36,850 

1.02 

4-year non-doctorate-granting 

297,512 

1.01 

4-year doctorate-granting 

191,197 

0.98 

Private for-profit 

Less-than-2-year 

221,174 

1.01 

2 years or more 

224,368 

1.06 

Fall full-time freshmen enrollment, by institution type 

Public 

Less-than-2-year 

25,855 

1.05 

2-year 

628,624 

0.99 

4-year non-doctorate-granting 

292,891 

0.99 

4-year doctorate-granting 

587,041 

1.00 

Private nonprofit 

Less than 4-year 

32,490 

1.02 

4-year non-doctorate-granting 

282,847 

1.01 

4-year doctorate-granting 

186,294 

0.98 

Private for-profit 

Less-than-2-year 

196,685 

1.02 

2 years or more 

196,903 

1.06 

Fall full-time freshmen receiving loans, by institution type 

Public 

Less-than-2-year 

6,798 

1.07 

2-year 

106,245 

0.97 

4-year non-doctorate-granting 

131,000 

1.00 

4-year doctorate-granting 

258,939 

1.01 

Private nonprofit 

Less than 4-year 

14,082 

1.07 

4-year non-doctorate-granting 

174,260 

1.02 

4-year doctorate-granting 

98,419 

0.99 


See notes at end of table. 
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Table 39. Control totals and sums of weights for eligible cases for the BPS:04/09 panel weight 
raking adjustment: 2009 — Continued 


Model predictor variables 

Control total 

Panel weights - 
average weight 
adjustment factor 

Private for-profit 

Less-than-2-year 

156,147 

1.02 

2 years or more 

184,539 

1.03 

Amount of loans for fall full-time freshmen, by institution type 

Public 

Less-than-2-year 

$ 30,812,263 

1.07 

2 -year 

287,418,487 

0.97 

4-year non-doctorate-granting 

456,981,660 

1.00 

4-year doctorate-granting 

958,683,761 

1.01 

Private nonprofit 

Less than 4-year 

63,879,653 

1.07 

4-year non-doctorate-granting 

780,073,727 

1.02 

4-year doctorate-granting 

500,662,774 

0.99 

Private for-profit 

Less-than-2-year 

670,258,172 

1.02 

2 years or more 

1,046,332,274 

1.03 

Fall full-time freshmen receiving federal grants, by institution type 

Public 

Less-than-2-year 

18,573 

1.04 

2 -year 

244,510 

0.97 

4-year non-doctorate-granting 

104,860 

1.02 

4-year doctorate-granting 

152,073 

1.01 

Private nonprofit 

Less than 4-year 

24,491 

1.00 

4-year non-doctorate-granting 

100,945 

1.04 

4-year doctorate-granting 

42,537 

1.00 

Private for-profit 

Less-than-2-year 

192,864 

1.04 

2 years or more 

156,518 

0.99 

Amount of federal grants for fall full-time freshmen, by institution type 

Public 

Less-than-2-year 

$45,434,148 

1.04 

2 -year 

667,666,709 

0.97 

4-year non-doctorate-granting 

314 , 444,744 

1.02 

4-year doctorate-granting 

459,752,080 

1.01 

Private nonprofit 

Less than 4-year 

79,273,524 

1.00 

4-year non-doctorate-granting 

315,420,406 

1.04 

4-year doctorate-granting 

157,065,462 

1.00 

Private for-profit 

Less-than-2-year 

570,173,299 

1.04 

2 years or more 

432,959,930 

0.99 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


Table 40 summarizes the weight distributions and the variance inflation due to unequal 
weighting by type of institution. The median student study weight ranges from 73 for students 
whose base year institution was private nonprofit less than 4-year to 236 for students whose base 
year institution was private for-profit 2 years or more. The mean student study weight ranges from 
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102 for students whose base year institution was private nonprofit less than 4-year to 325 for 
students whose base year institution was private-for-profit 2 years or more. The unequal weighting 
effect overall is 2.22, and ranges from 1.40 for students whose base year institution was public 4- 
year doctorate-granting to 3.37 for students whose base year institution was private nonprofit less 
than 4-year. 


Table 40. Weight distribution and unequal weighting effects for the NPSAS:04 - BPS:04/06 - 
BPS:04/09 longitudinal study weight, by type of institution: 2009 


Type of institution 

Minimum 

First 

quartile 

Median 

Third 

quartile 

Maximum 

Mean 

Unequal 

weighting 

effect 

Total 

0.91 

96.20 

169.09 

277.18 

5299.73 

232.36 

2.22 

Institutional sector 

Public 

Less-than-2-year 

6.32 

39.25 

76.75 

145.09 

1324.78 

128.99 

2.58 

2-year 

2.44 

119.58 

204.35 

348.28 

5299.73 

301.91 

2.24 

4-year non-doctorate- 

granting 

0.91 

111.98 

213.56 

311.41 

1975.74 

239.44 

1.64 

4-year doctorate-granting 

3.12 

126.62 

216.20 

265.98 

1611.54 

211.25 

1.40 

Private nonprofit 

Less than 4-year 

1.31 

40.85 

73.44 

119.44 

2301.75 

102.00 

3.37 

4-year non-doctorate- 

granting 

1.22 

72.34 

133.05 

193.77 

1744.19 

155.90 

1.80 

4-year doctorate-granting 

1.23 

78.50 

115.18 

146.16 

2817.56 

128.86 

1.79 

Private for-profit 

Less-than-2-year 

1.19 

27.46 

153.58 

340.11 

2937.87 

243.95 

2.62 

2 years or more 

2.85 

133.83 

235.87 

381.02 

3073.49 

324.86 

2.07 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


To assess the overall predictive ability of the nonresponse model, an ROC curve, described 
in section 6.1.1, was used to provide a measure of how well the model correctly classified 
individuals of known response type. The plot of the first probability against the second (that is, the 
proportion of respondents with a predicted probability of response greater than c versus the 
proportion of nonrespondents with a predicted probability of response greater than c), for c 
ranging from 0 to 1, resulted in the ROC curve shown in figure 23. This figure shows that the area 
under the ROC curve is 0.74 such that 74 percent of the time (or more than 7 of 10 pairings) the 
predicted probabilities give the correct classification. The Wilcoxon test (T= 0.74) result fails to 
support the null hypothesis of no predictive ability (p < 0.05). This level of discrimination implies 
that the variables used in the model are highly informative but not definite predictors of a sample 
student’s overall response propensity. 
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Figure 23. Receiver operating characteristics (ROC) curve for NPSAS:04 - BPS:04/06 - BPS:04/09 
longitudinal study response propensity: 2009 



Pfresponse | respondent) 


Observed ROC- area under the curve = 0.73731 
50% predictive ROC 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

6.1.3 Analysis Weights for Cases with Transcript Data 

A weight was also constructed for analyzing the cases with transcript data. Of the 18,640 
students who were eligible for BPS:04/06, 110 were deceased, 16,960 had some transcript data and 
met the definition of a “transcript respondent,” and the remaining 1,580 were considered 
nonrespondents for this weight. 

As with the weights described in sections 6.1.1 and 6.1.2, the initial weight was the 
BPS:04/ 06 analysis weight. An adjustment was made for nonresponse using a model-based 
constrained logistic weighting procedure, then the weights were calibrated to the sums of the 
BPS:04/09 study weights for eligible cases. The procedure WTADJUST in SUDAAN was used to 
implement the nonresponse and calibration adjustments. 

The first adjustment was for nonresponse, that is, not having transcript data. The 
adjustment model included the 18,540 cases who were not deceased, with the response indicator set 
to 1 for the 16,960 cases with transcript data and set to 0 for the 1,580 cases who were nondeceased 
transcript nonrespondents. Predictor variables were chosen if considered to be predictive of 
response status and were nonmissing for both study respondents and nonrespondents. Variables 
used in the nonresponse adjustment models for NPSAS and BPS were also included. Candidate 
predictor variables included the same set of variables that was used for the study weight (see section 
6 . 1 . 1 ). 
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Variables included in the nonresponse modeling included all of the candidate predictor 
variables as well as certain important interaction terms identified using. CHAID was run for up to 
three segments, resulting in the identification of two-way and three-way interactions. Seven 
variables that made up the CHAID interaction terms for the student transcript weight adjustment 
were: the state in which the base year institution is located, type of institution, whether the student 
was a BPS:04/ 06 interview respondent, retention and attainment through 2006, base year 
institution undergraduate enrollment, race/ethnicity, and type of aid package. 

Table 41 shows the predictor variables used in the model to adjust the weight and the 
average weight adjustment factors resulting from these variables. The nonresponse weight 
adjustment factors have the following characteristics: 

• Minimum: 1.00; 

• Median: 1.01; and 

• Maximum: 3.45 
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Table 41. Weight adjustment factors for nonresponse for the BPS:04/09 student transcript weight: 
2009 


Average 

Weighted weight 

Number of response adjustment 

Model predictor variables respondents rate factor 


Total 

16,960 

91.42 

1.09 

Type of institution 

Public 

Less-than-2-year 

400 

74.74 

1.29 

2-year 

5,840 

91.68 

1.08 

4-year non-doctorate-granting 

1,540 

90.41 

1.09 

4-year doctorate-granting 

2,940 

95.94 

1.04 

Private nonprofit 

Less than 4-year 

480 

89.00 

1.12 

4-year 

3,650 

96.25 

1.03 

Private for-profit 

2,110 

82.63 

1.27 

NPSAS institution control 2003-04 

Public 

10,720 

92.18 

1.08 

Private nonprofit 

4,130 

95.71 

1.04 

Private for-profit 

2,110 

82.63 

1.27 

NPSAS institution level 2003-04 

Less-than-2-year 

1,550 

74.26 

1.36 

2-year 

6,880 

91.31 

1.09 

4-year 

8,530 

94.54 

1.05 

In 12 state NPSAS:04 representative sample 

Yes 

8,610 

92.57 

1.07 

No 

8,340 

90.58 

1.12 

Bureau of Economic Analysis code (Office of Business Economics 
[OBE]) Region 

New England (CT ME MA NH Rl VT) 

1,100 

90.90 

1.10 

Mid East (DE DC MD NJ NY PA) 

2,470 

93.74 

1.06 

Great Lakes (IL IN Ml OHWI) 

2,920 

91.62 

1.09 

Plains (IAKS MN MO NE ND SD) 

1,700 

86.18 

1.11 

SE (ALAR FL GA KY LA MS NO SC TN VA WV) 

3,840 

91.20 

1.09 

Southwest (AZ NM OKTX) 

1,910 

94.31 

1.08 

Rocky Mountains (CO ID MT UT WY) 

600 

96.71 

1.05 

Far West (AK CA HI NV OR WA) 

2,160 

87.91 

1.14 

Other jurisdictions (PR) 

270 

99.59 

1.01 

Institution enrollment size 

1,821 or fewer 

3,920 

85.82 

1.18 

1,822 to 6,690 

4,290 

91.95 

1.06 

6,691 to 16,522 

4,310 

91.24 

1.10 

More than 16,522 

4,430 

94.92 

1.03 

Age as of 12/31/03 

15-18 

7,890 

93.53 

1.06 

19 

4,330 

92.44 

1.08 

20-23 

1,900 

88.50 

1.15 

24-29 

1,150 

89.25 

1.15 

30 or older 

1,690 

86.63 

1.18 


See notes at end of table. 
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Chapter 6. Weighting and Variance Estimation 


Table 41 . Weight adjustment factors for nonresponse for the BPS:04/09 student transcript 
weight: 2009 — Continued 

Average 

Weighted weight 

Number of response adjustment 

Model predictor variables respondents rate factor 

Gender 


Male 

7,090 

91.90 

1.09 

Female 

9,870 

91.07 

1.10 

Race/ethnicity 

White 

10,820 

92.25 

1.08 

Black or African American 

2,200 

89.46 

1.12 

Hispanic or Latino 

2,260 

89.45 

1.14 

Asian 

780 

90.51 

1.11 

American Indian or Alaska Native, Native Hawaiian or other 

Pacific Islander, Other, More than one race 

890 

93.47 

1.07 

Citizenship status, 2003-04 

US citizen 

16,040 

91.59 

1.09 

Resident alien 

730 

87.30 

1.17 

Foreign or international student 

190 

95.72 

1.05 

Number of institutions attended 2003-04 (NPSAS) 

1 

15,670 

91.10 

1.10 

2 or 3 

1,290 

97.03 

1.02 

Enrollment pattern 2003-04 (NPSAS) 

Enrolled mostly full-time 

14,090 

92.05 

1.09 

Enrolled mostly part-time 

2,480 

89.67 

1.11 

Enrolled full-time & part-time equally 

390 

89.52 

1.10 

Any dependent children 2003-04 (NPSA) 

Yes 

2,490 

85.85 

1.19 

No 

14,470 

92.62 

1.08 

Number of family members in college for independent student, or 
dependent student in 2003-04 

1 

3,660 

87.91 

1.16 

2 

330 

84.85 

1.22 

3 or 4 

40 

90.95 

1.14 

Dependent student 

12,920 

92.92 

1.07 

Total income by dependency, 2003-04 

Dependent student 

Less than $10,000 

730 

91.42 

1.08 

$10,000-$1 9,999 

1,070 

92.51 

1.07 

$20,000-$29,999 

1,320 

89.92 

1.11 

$30,000-$39,999 

1,300 

91.91 

1.09 

$40,000-$49,999 

1,170 

94.67 

1.06 

$50,000-$59,999 

1,080 

92.22 

1.08 

$60,000-$69,999 

1,190 

91.49 

1.09 

$70,000-$79,999 

910 

92.87 

1.07 

$80,000-$99,999 

1,550 

94.23 

1.05 

$100,000 or more 

2,610 

95.21 

1.04 


See notes at end of table. 
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Chapter 6. Weighting and Variance Estimation 


Table 41 . Weight adjustment factors for nonresponse for the BPS:04/09 student transcript 
weight: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Independent student 

Less than $10,000 

770 

87.58 

1.14 

$10,000-$1 9,999 

1,450 

86.93 

1.17 

$20,000-$29,999 

670 

87.11 

1.17 

$30,000-$49,999 

630 

88.20 

1.17 

$50,000 or more 

510 

89.30 

1.15 

Degree plans first year, 2003-04 

Certificate 

2,320 

81.66 

1.25 

Associate's degree 

5,440 

91.35 

1.09 

Bachelor's degree 

7,890 

94.40 

1.05 

No degree 

1,310 

91.95 

1.08 

Degree types attained through 2006 

Certificate only, associate’s only, or certificate and associate’s 

2,640 

87.78 

1.17 

Associate’s and bachelor’s or bachelor's only 

210 

93.13 

1.06 

Never attained 

14,100 

92.04 

1.08 

Highest degree attained anywhere, 2006 

Certificate 

1,380 

82.13 

1.26 

Associate's 

1,260 

93.78 

1.07 

Bachelor's degree 

210 

93.13 

1.06 

Never attained 

14,100 

92.04 

1.08 

Persistence and attainment anywhere, 2006 

Attained, still enrolled 

1,240 

93.81 

1.07 

Attained, not enrolled 

1,620 

83.46 

1.23 

No degree, still enrolled 

9,150 

93.75 

1.05 

No degree, not enrolled 

4,960 

89.39 

1.12 

Retention and attainment at first institution 3-year total 2006 

Bachelor's degree 

200 

92.69 

1.07 

Associate’s degree 

1,100 

93.10 

1.08 

Certificate 

1,250 

81.45 

1.27 

No degree, still enrolled 

7,470 

91.98 

1.07 

No degree, transferred 

3,350 

97.31 

1.02 

No degree, left without return 

3,600 

88.67 

1.14 

Parent's highest education level, 2003-04 

Did not complete high school 

1,110 

85.90 

1.18 

High school diploma or equivalent 

4,380 

89.52 

1.12 

Vocational or technical training 

680 

93.22 

1.07 

Less than two years of college 

1,310 

92.19 

1.07 

Associate's degree 

1,280 

93.42 

1.07 

2 or more years of college but no degree 

890 

89.89 

1.12 

Bachelor's degree 

3,580 

93.59 

1.06 

Master's degree or equivalent 

2,290 

93.87 

1.06 

First-professional degree 

420 

95.15 

1.04 

Doctoral degree or equivalent 

630 

92.66 

1.07 

Do not know parent's education level 

400 

87.43 

1.17 


See notes at end of table. 
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Chapter 6. Weighting and Variance Estimation 


Table 41 . Weight adjustment factors for nonresponse for the BPS:04/09 student transcript 
weight: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Applied for any aid, 2003-04 

Yes 

14,940 

91.26 

1.09 

No 

2,010 

92.30 

1.09 

Aid package by type of aid, 2003-04 

Grants only 

4,750 

91.61 

1.09 

Loans only 

970 

91.27 

1.13 

Work-study only or Work-study and other 

110 

97.32 

1.02 

Other only 

200 

94.23 

1.10 

Grants and loans 

3,600 

89.99 

1.11 

Grants and work-study 

520 

94.16 

1.04 

Grants and other 

250 

92.30 

1.09 

Loans and work-study 

80 

94.00 

1.05 

Loans and other 

320 

93.26 

1.11 

Grants, loans, and work-study 

1,190 

95.13 

1.04 

Grants, loans, and other 

780 

87.55 

1.13 

Grants, work-study, and other 

20 

82.59 

1.27 

Loans, work-study, and other with and without grants 

340 

94.09 

1.07 

No aid received 

3,830 

91.46 

1.09 

Pell Grant recipient 

Yes 

6,260 

89.32 

1.12 

No 

10,700 

92.59 

1.08 

Pell Grant amount received (in dollars) 

None 

10,700 

92.59 

1.08 

$2,000 or less 

2,070 

89.43 

1.11 

$2,001 to $3,700 

2,170 

89.44 

1.12 

More than $3,700 

2,020 

89.03 

1.12 

Stafford Loan recipient 

Yes 

6,870 

90.70 

1.10 

No 

10,090 

91.81 

1.09 

Stafford Loan amount received (in dollars) 

None 

10,090 

91.81 

1.09 

$2,624 or less 

1,190 

89.29 

1.10 

$2625 

3,570 

94.25 

1.06 

More than $2,625 

2,110 

86.61 

1.17 

Federal aid recipient 

Yes 

10,350 

90.83 

1.10 

No 

6,610 

92.14 

1.08 

Federal aid amount received (in dollars) 

None 

6,610 

92.14 

1.08 

$2,831 or less 

3,450 

92.16 

1.09 

$2,832 to $6,625 

3,550 

90.33 

1.10 

More than $6,625 

3,340 

89.77 

1.11 

Institutional aid recipient 

Yes 

5,250 

94.44 

1.05 

No 

11,710 

90.40 

1.11 


See notes at end of table. 
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Chapter 6. Weighting and Variance Estimation 


Table 41 . Weight adjustment factors for nonresponse for the BPS:04/09 student transcript 
weight: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

State aid recipient 

Yes 

4,120 

94.55 

1.05 

No 

12,830 

90.60 

1.10 

PLUS amount received (in dollars) 

None 

15,540 

91.50 

1.09 

Less than $5,825 

460 

85.31 

1.12 

$5,825 to $10,207 

490 

91.92 

1.10 

More than $10,207 

470 

94.46 

1.07 

Data available from Central Processing System (CPS), 2003-04 

Yes 

12,470 

91.15 

1.10 

No 

4,480 

92.02 

1.08 

In BPS:04/09 field cluster 

Yes 

1,210 

85.04 

1.18 

No 

15,740 

92.00 

1.09 

NPSAS:04 interview respondent 

Yes 

16,620 

91.63 

1.09 

No 

340 

87.89 

1.16 

BPS:04/06 interview respondent 

Yes 

13,700 

92.24 

1.08 

No 

3,260 

88.60 

1.14 

Chi-square automatic interaction detection (CHAID) segments 

Base year institution located in AR, AZ, NY, TX and is public 2- 
year, public 4-year non-doctorate-granting, or public 4-year 
doctorate-granting 

1,840 

98.13 

1.02 

Base year institution located in WA, LA, NE, OR, UT and is 

public 2-year, public 4-year non-doctorate-granting, or public 
4-year doctorate-granting 

970 

94.96 

1.04 

Base year institution located in DE or PA and is public 2-year, 
public 4-year non-doctorate-granting, or public 4-year 
doctorate-granting 

420 

99.76 

1.00 

Base year institution located in AR, AZ, PA, NY, WA, DE, LA, NE, 
OR, TX, or UT 

Institution type is public less-than-2-year, private nonprofit less 
than 4-year, private nonprofit non-doctorate-granting, or 
private for-profit 2 years or more 

1,190 

99.05 

1.01 

Institution type is private nonprofit 4-year doctorate-granting 
and undergraduate enrollment is less than 4,178 

140 

97.05 

1.03 

Institution type is private nonprofit 4-year doctorate- granting 
and undergraduate enrollment is between 4,178 and 

6,646 

60 

98.01 

1.02 

Institution type is private nonprofit 4-year doctorate-granting 
and undergraduate enrollment is more than 6,646 

400 

98.64 

1.01 

Base year institution located in AR, WA, DE, LA, NE, TX and is 

private for-profit less-than-2-year 

280 

84.51 

1.19 

Base year institution located in AZ, NY, OR, PA, UT and is 

private for-profit less-than-2-year 

110 

50.10 

2.07 

See notes at end of table. 
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Chapter 6. Weighting and Variance Estimation 


Table 41 . Weight adjustment factors for nonresponse for the BPS:04/09 student transcript 
weight: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Base year institution located in FL or IA 

1,120 

94.30 

1.06 

Base year institution located in TN, OH, CO, MD, IL, Wl, MA, Rl 
and undergraduate enrollment 593 or fewer students 

300 

63.94 

1.49 

Base year institution located in CO, IL, MA, OH, MD, Wl, Rl, TN 
and undergraduate enrollment is between 593 and 4,178 
students 

960 

92.82 

1.07 

Base year institution located in OH, CO, MD, IL, Wl, MA, Rl and 
undergraduate enrollment is between 4,178 and 6,646 
students 

330 

97.30 

1.02 

Base year institution located in TN or CO and undergraduate 
enrollment is between 6,646 and 9,971 students 

70 

77.53 

1.25 

Base year institution located in IL, OH, MD, MA, Rl, Wl and 
undergraduate enrollment is between 6,646 and 9,971 
students 

240 

84.55 

1.17 

Base year institution located in TN, OH, CO, MD, IL, Wl, MA, Rl 
and undergraduate enrollment between 9,971 and 14,068 

310 

92.88 

1.06 

Base year institution located in TN, OH, CO, MD, IL, Wl, MA, Rl 
and undergraduate enrollment more than 14,068 students, or 
base year institution located in ID, SD, NM, WY, MT, AK, VT 

1,370 

98.03 

1.02 

Base year institution located in KS, NJ, CA, OK, AL, CT, KY, OK 
Undergraduate enrollment less than or equal to 593 and aid 
package is grants, loans, work study; loans only; loans 
and work study; grants and work study; or loans and other 
type of aid 

70 

95.12 

1.06 

Undergraduate enrollment less than or equal to 593 and aid 
package is no aid; grants only; other aid only; grants, 
works study and other aid; grants, loans, or grants, loans, 
work study, and other aid 

80 

47.23 

2.06 

Undergraduate enrollment less than or equal to 593 and other 
types of aid package 

140 

67.03 

1.51 

Undergraduate enrollment between 593 and 1,485 

280 

73.45 

1.38 

Undergraduate enrollment between 1,485 and 2,406 

250 

99.22 

1.01 

Undergraduate enrollment between 2,406 and 4,178 

330 

93.70 

1.07 

Undergraduate enrollment between 4,178 and 14,068 

730 

79.14 

1.24 

Undergraduate enrollment between 14,068 and 28,881 

610 

93.53 

1.05 

Undergraduate enrollment more than 28,881 and 
race/ethnicity is white or black 

70 

71.45 

1.34 

Undergraduate enrollment more than 28,881 and 
race/ethnicity other than white or black 

130 

92.72 

1.06 

Base year institution located in MO or WV 

Undergraduate enrollment less than or equal 4,178 students 

130 

88.50 

1.11 

Undergraduate enrollment more than 4,178 students 

160 

53.27 

1.88 


See notes at end of table. 
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Chapter 6. Weighting and Variance Estimation 


Table 41 . Weight adjustment factors for nonresponse for the BPS:04/09 student transcript 
weight: 2009 — Continued 


Model predictor variables 

Number of 
respondents 

Weighted 

response 

rate 

Average 

weight 

adjustment 

factor 

Base year institution located in VA, MS, DC, ND 

Undergraduate enrollment less than or equal 6,646 

310 

83.35 

1.21 

Undergraduate enrollment more than 6,646 and not enrolled in 
2006 

60 

46.30 

1.91 

Undergraduate enrollment more than 6,646 and enrolled in 

2006 

150 

67.48 

1.53 

Base year institution located in PR, GA, MN, SC, HI 

Undergraduate enrollment less than or equal to 593 

110 

95.11 

1.05 

Undergraduate enrollment more than 593 

1,740 

99.79 

1.00 

Base year institution located in IN, Ml, NV, NC, NH, ME 
Undergraduate enrollment less than or equal to 593 and 
BPS:04/06 interview respondent 

110 

95.88 

1.05 

Undergraduate enrollment less than or equal to 593 and 
BPS:04/06 interview nonrespondent 

50 

71.29 

1.37 

Undergraduate enrollment between 593 and 2,406 

260 

99.68 

1.00 

Undergraduate enrollment between 2,406 and 14,068 

410 

98.99 

1.01 

Undergraduate enrollment more than 14,068 

690 

99.55 

1.01 


NOTE: Detail may not sum to totals because of rounding. Categories of undergraduate enrollment were determined by quartiles. 
Categories of Pell grant amount and Stafford loan amount were determined by tertiles. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

To ensure population coverage and consistency with the BPS04/09 study weight, and the 
BPS:04/ 06 and NPSAS:04 weights, the BPS:04/ 09 panel transcript was adjusted to control totals 
determined by the BPS:04/ 09 study weight sums. Cases which were deceased were not included in 
either the control totals or in the cases included in the adjustment. This adjustment was also 
implemented using the SUDAAN WTADJUST procedure. Variables used to define the control 
totals were the same as those used for the poststratification coverage adjustments for the 
BPS:04/ 09 study weight. The control totals for the BPS:04/ 09 transcript weights were established 
by the weighted sums from the BPS:04/09 study weights. 

Table 42 gives the variables used for the calibration, the values of the control totals, and the 
average weight adjustment factors for each variable. Statistics for the weight adjustment factors are 
the following: 

• Minimum: 0.67, 

• Median: 1 .00, and 

• Maximum: 1 .23 

The response adjusted, calibrated transcript weight is the variable WTC000 on the data file. 
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Chapter 6. Weighting and Variance Estimation 


Table 42. Control totals and sums of weights for eligible cases for the BPS:04/09 student 
transcript weight raking: 2009 



Control total 

Panel weights -- 
average weight 
adjustment factor 

Total 

3,746,295 

1.00 

Total fall undergraduate enrollment 

3,199,574 

1.00 

Fall enrollment, by institution type 

Public 

Less-than-2-year 

43,700 

0.99 

2-year 

1,242,966 

1.00 

4-year non-doctorate-granting 

333,965 

0.98 

4-year doctorate-granting 

623,377 

1.01 

Private nonprofit 

Less than 4-year 

34,623 

0.97 

4-year non-doctorate-granting 

298,554 

1.00 

4-year doctorate-granting 

198,107 

1.00 

Private for-profit 

Less-than-2-year 

200,697 

0.99 

2 years or more 

223,583 

1.00 

Non-fall enrollment, by institution type 

Public 

Less-than-2-year 

43,378 

0.99 

2-year 

1,505,878 

1.00 

4-year non-doctorate-granting 

344,941 

0.98 

4-year doctorate-granting 

614,269 

1.01 

Private nonprofit 

Less than 4-year 

36,717 

0.97 

4-year non-doctorate-granting 

319,078 

1.00 

4-year doctorate-granting 

194,066 

1.00 

Private for-profit 

Less-than-2-year 

208,671 

0.99 

2 years or more 

263,162 

1.00 

Amount of Pell Grants awarded, by institution type (in dollars) 

Public 

Less-than-2-year 

$ 53,793,325 

1.00 

2-year 

987,176,888 

1.00 

4-year non-doctorate-granting 

326,046,179 

0.96 

4-year doctorate-granting 

420,302,200 

1.02 

Private nonprofit 

Less than 4-year 

82,072,863 

0.99 

4-year non-doctorate-granting 

273,647,143 

0.99 

4-year doctorate-granting 

115,766,595 

1.01 

Private for-profit 

Less-than-2-year 

598,631,871 

1.01 

2 years or more 

492,562,087 

1.01 

Amount of Stafford Loans awarded, by institution type (in dollars) 

Public 

Less-than-2-year 

$ 40,072,780 

0.92 

2-year 

401,473,395 

1.00 

4-year non-doctorate-granting 

452,192,315 

0.98 

4-year doctorate-granting 

784,412,732 

1.01 


See notes at end of table. 
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Table 42. Control totals and sums of weights for eligible cases for the BPS:04/09 student 
transcript weight raking: 2009 — Continued 



Control total 

Panel weights — 
average weight 
adjustment factor 

Private nonprofit 

Less than 4-year 

70,300,924 

0.96 

4-year non-doctorate-granting 

610,392,641 

1.00 

4-year doctorate-granting 

313,355,130 

0.99 

Private for-profit 

Less-than-2-year 

665,887,187 

0.98 

2 years or more 

1,108,476,038 

1.01 

Number of fall freshmen, by institution type 

Public 

Less-than-2-year 

43,363 

0.99 

2-year 

1,039,308 

1.00 

4-year non-doctorate-granting 

320,880 

0.98 

4-year doctorate-granting 

607,943 

1.01 

Private nonprofit 

Less than 4-year 

36,850 

0.97 

4-year non-doctorate-granting 

297,512 

1.00 

4-year doctorate-granting 

191,197 

1.00 

Private for-profit 

Less-than-2-year 

221,174 

0.99 

2 years or more 

224,368 

0.99 

Fall full-time freshmen enrollment, by institution type 

Public 

Less-than-2-year 

25,855 

0.99 

2-year 

628,624 

0.99 

4-year non-doctorate-granting 

292,891 

0.98 

4-year doctorate-granting 

587,041 

1.01 

Private nonprofit 

Less than 4-year 

32,490 

0.97 

4-year non-doctorate-granting 

282,847 

1.00 

4-year doctorate-granting 

186,294 

1.00 

Private for-profit 

Less-than-2-year 

196,685 

0.98 

2 years or more 

196,903 

0.99 

Fall full-time freshmen receiving loans, by institution type 

Public 

Less-than-2-year 

6,798 

0.91 

2-year 

106,245 

1.00 

4-year non-doctorate-granting 

131,000 

0.97 

4-year doctorate-granting 

258,939 

1.01 

Private nonprofit 

Less than 4-year 

14,082 

0.95 

4-year non-doctorate-granting 

174,260 

1.00 

4-year doctorate-granting 

98,419 

0.99 

Private for-profit 

Less-than-2-year 

156,147 

0.97 

2 years or more 

184,539 

0.99 


See notes at end of table. 
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Table 42. Control totals and sums of weights for eligible cases for the BPS:04/09 student 
transcript weight raking: 2009 — Continued 



Control total 

Panel weights - 
average weight 
adjustment factor 

Amount of loans for fall full-time freshmen, by institution type 

Public 

Less-than-2-year 

$ 30,812,263 

0.91 

2 -year 

287,418,487 

1.00 

4-year non-doctorate-granting 

456,981,660 

0.97 

4-year doctorate-granting 

958,683,761 

1.01 

Private nonprofit 

Less than 4-year 

63,879,653 

0.95 

4-year non-doctorate-granting 

780,073,727 

1.00 

4-year doctorate-granting 

500,662,774 

0.99 

Private for-profit 

Less-than-2-year 

670,258,172 

0.97 

2 years or more 

1,046,332,274 

0.99 

Fall full-time freshmen receiving federal grants, by institution type 

Public 

Less-than-2-year 

18,573 

1.00 

2 -year 

244,510 

1.00 

4-year non-doctorate-granting 

104,860 

0.96 

4-year doctorate-granting 

152,073 

1.01 

Private nonprofit 

Less than 4-year 

24,491 

0.99 

4-year non-doctorate-granting 

100,945 

0.98 

4-year doctorate-granting 

42,537 

1.00 

Private for-profit 

Less-than-2-year 

192,864 

1.00 

2 years or more 

156,518 

1.00 

Amount of federal grants for fall full-time freshmen, by institution type 

Public 

Less-than-2-year 

$ 45 , 434,148 

1.00 

2 -year 

667,666,709 

1.00 

4-year non-doctorate-granting 

314 , 444,744 

0.96 

4-year doctorate-granting 

459,752,080 

1.01 

Private nonprofit 

Less than 4-year 

79,273,524 

0.99 

4-year non-doctorate-granting 

315,420,406 

0.98 

4-year doctorate-granting 

157,065,462 

1.00 

Private for-profit 

Less-than-2-year 

570,173,299 

1.00 

2 years or more 

432,959,930 

1.00 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


Table 43 summarizes the weight distributions and the variance inflation due to unequal 
weighting by type of institution. The median student study weight ranges from 63 for students 
whose base year institution was private nonprofit less than 4-year to 219 for students whose base 
year institution was public 4-year doctorate-granting. The mean student study weight ranges from 
87 for students whose base year institution was private nonprofit less than 4-year to 277 for 
students whose base year institution was public 2-year. The unequal weighting effect overall is 2.01, 
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and ranges from 1.33 for students whose base year institution was public 4-year doctorate -granting 
to 2.77 for students whose base year institution was public less-than-2-year. 


Table 43. Weight distribution and unequal weighting effects for the BPS:04/09 student transcript 
weight, by type of institution: 2009 


Type of institution 

Minimum 

First 

quartile 

Median 

Third 

quartile 

Maximum 

Mean 

Unequal 

weighting 

effect 

Total 

0.85 

92.45 

164.48 

269.91 

4264.70 

220.96 

2.01 

Institutional sector 

Public 

Less-than-2-year 

2.63 

38.71 

67.96 

135.73 

1292.43 

119.53 

2.77 

2-year 

2.06 

112.55 

194.74 

324.79 

4264.70 

276.86 

2.04 

4-year non-doctorate-granting 

0.85 

110.86 

202.11 

313.91 

1722.45 

241.41 

1.60 

4-year doctorate-granting 

3.15 

129.18 

218.77 

271.90 

1434.43 

213.21 

1.33 

Private nonprofit 

Less than 4-year 

1.28 

35.36 

63.04 

104.64 

1367.86 

87.12 

2.69 

4-year non-doctorate-granting 

1.03 

70.62 

135.68 

194.79 

1676.03 

154.62 

1.78 

4-year doctorate-granting 

1.55 

79.98 

117.67 

152.19 

1707.31 

130.41 

1.53 

Private for-profit 

Less-than-2-year 

1.29 

25.68 

138.95 

315.84 

3353.31 

216.53 

2.45 

2 years or more 

2.43 

119.18 

200.58 

333.14 

3289.90 

270.63 

1.95 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


To assess the overall predictive ability of the nonresponse model, an ROC curve was again 
used to provide a measure of how well the model correctly classified individuals of known response 
type. The plot of the first probability against the second (that is, the proportion of respondents 
with a predicted probability of response greater than c versus the proportion of nonrespondents 
with a predicted probability of response greater than c), for c ranging from 0 to 1, resulted in the 
ROC curve shown in figure 24. The area under the ROC curve is 0.66, such that 66 percent of the 
time (or almost 7 of 10 pairings), the predicted probabilities give the correct classification. The 
ROC area of 0.66 equals the value of the Wilcoxon test statistic; the Wilcoxon test fails to support 
the null hypothesis of no predictive ability (p < 0.05). This level of discrimination implies that the 
variables used in the model are highly informative but not definite predictors of a sample student’s 
overall response propensity. 
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Figure 24. Receiver operating characteristics (ROC) curve for BPS:04/09 transcript response 
propensity: 2009 



P(response | respondent) 


Observed ROC- area under the curve = 0.65583 
50% predictive ROC 


SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

6.2 Variance Estimation 

For probability-based sample surveys, most estimates are nonlinear statistics. For example, 
the estimate of a mean or proportion, which is expressed as 

Y w i x i 

m = ^ , 

i 

is nonlinear because the denominator is a survey estimate of the (unknown) population total. In this 
situation, the variances of the estimates cannot be expressed in closed form. Two procedures for 
estimating variances of survey statistics are the Taylor series linearization procedure and the 
bootstrap replication procedure. Variables to use for both of these variance estimation procedures 
are available on the BPS:04/09 data files. The analysis strata and replicates created for the Taylor 
series procedure are discussed in section 6.2.1, and section 6.2.2 discusses the replicate weights 
created for the bootstrap procedure. 

6.2.1 Taylor Series 

The Taylor series variance estimation procedure is a well-known technique used to estimate 
the variances of nonlinear statistics. The procedure takes the first-order Taylor series approximation 
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of the nonlinear statistic and substitutes the linear representation into the appropriate variance 
formula based on the sample design. Woodruff (1971) presented the mathematical formulation of 
this procedure. NPSAS:04 provided two sets of variables for Taylor series variance estimation, and 
BPS:04/ 09 also provides two sets of variables. One set of variables is used in software that assumes 
that the first stage sampling units (institutions) were sampled with replacement, and does not 
account for the finite population correction (FPC) at the institution level of sampling. The other set 
of variables is used in software that will account for without replacement sampling of institutions in 
the calculation of variances, and does account for the FPC. Both sets of variables are provided 
because not all survey data analysis packages have the option to incorporate the FPC in the 
variance calculations. When the first stage units are sampled with very small probabilities, the 
estimated variances using the with replacement variance formulas and the without replacement 
variance formulas are almost the same, but in NPSAS:04, some institutions were sampled with high 
sampling probabilities. 

The first set of variables described assumes that the first stage units were sampled with 
replacement (or with small selection probabilities). For stratified multistage surveys, the Taylor 
series procedure requires variance estimation strata and variance estimation primary sampling units 
(PSUs), also called replicates , defined from the sampling strata and PSUs used in the first stage of 
sampling. Because BPS:04/ 09 is a follow-up study of both NPSAS:04 and BPS:04/ 06, the variance 
estimation strata and PSUs for BPS:04/06 and BPS:04/09 were derived from the variance 
estimation strata and PSUs that were developed for NPSAS:04. The steps in the construction of the 
NPSAS:04 and BPS:04/ 06 stratum and PSU variables are described in chapter 6 of the NPSAS:04 
Full-scale Methodology Report (Cominole et al. 2006) and chapter 6 of the BPS:04/ 06 Full-scale 
Methodology Report (Cominole et al. 2007). 

The variance estimation formulas require at least two PSUs in each stratum. The NPSAS:04 
variance estimation strata and PSUs were examined for the BPS:04/09 sample, and strata with only 
one PSU were combined with other strata to obtain at least two PSUs. The following three rules 
were used: variance estimation strata were combined with other variance estimation strata within 
the original NPSAS:04 sampling strata, certainty schools were combined with other certainty 
schools, and noncertainty schools were combined with other noncertainty schools. In addition, the 
original sort order that was used for constructing the NPSAS:04 variance estimation strata and 
PSUs was used. A variance estimation stratum was combined with the next stratum in the sorted 
list. If the stratum was the first in the sorted list, then it was combined with the next stratum in the 
list. The single PSU then became an additional PSU in the new variance estimation strata. The 
resulting variance estimation strata and PSUs for BPS:04/09 analyses using the study, panel, or 
transcript weight are the variables BPS09STR and BPS09PSU. 

The second set of variables that were created are to be used with software that allow for 
approximating variances using without replacement sampling and account for the FPC (which may 
reduce the estimate of the variance contribution at the first stage of sampling). The variables 
FB09STR, FB09PSU, FB09SSU and B9PSUCNT are comparable to the NPSAS:04 variables 
FANALSTR, FANALPSU, FANALSSU, and PSUCOUNT, and were constructed in a similar 
manner. Ideally, the BPS:04/ 09 versions of these variables would be identical to the NPSAS:04 
versions. In general, for certainty institutions, FB09STR equals the institutional sampling stratum, 
FANALPSU equals BPS09STR, and FANALSSU equals BPS09SSU. In the noncertainty 
institutions, FB09STR equals BPS09STR, FB09PSU equals BPS09PSU, and FB09SSU equals 
FANALSSU (which was created by randomly dividing the NPSAS:04 analysis PSUs into two parts). 
These variables are a by-product of the Kaufman (2004) methodology for the bootstrap variance 
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estimation weights (described in section 6.2.2), and the justification for using the without 
replacement variance formulas follows from the assumptions in that paper. Some values of the 
variance estimation strata, PSU, and SSU variables were combined in order to have at least 2 SSUs 
in each PSU, and at least 2 PSUs in each stratum. 

An alternate variance estimation method using bootstrap replicate weights is also provided 
for users of the BPS:04/09 data, as described in section 6.2.2. Table 44 summarizes the weight and 
variance estimation variables and how they are used in selected software packages that allow for 
Taylor series variance estimation (SUDAAN, Stata, the SAS survey data analysis procedures, and 
IBM SPSS Complex Samples) and bootstrap variance estimation (SUDAAN, Stata, the SAS survey 
data analysis procedures, and WesVar). Variance estimates and design effects given in appendices in 
this report were produced using the bootstrap replicate weights. 
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Table 44. Use of analysis weights, replicate weights, and variance estimation strata and primary 
sampling unit (PSU) variables available from BPS:04/09 in selected survey data analysis 
software: 2009 


Survey data analysis software 

BPS: 04/09 study 
respondents; NPSAS:04 and 
BPS:04/09 study respondents 

Study respondents to all 
three of NPSAS:04, 
BPS:04/06, and BPS:04/09 

Students with at least one 
transcript at BPS:04/09 

Analysis weight for estimates 

WTA000 

WTB000 

WTC000 

Taylor series variance estimation (with replacement) 

Variance estimation strata 

and PSU variables BPS09STR and BPS09PSU 

BPS09STR and BPS09PSU 

BPS09STR and BPS09PSU 


Software: statements, parameters, and keywords for Taylor series variance estimation (with replacement) 


SUDAAN 

DESIGN = WR 

WEIGHT WTA000; 

NEST BPS09STR 

BPS09PSU; 

DESIGN = WR 

WEIGHT WTB000; 

NEST BPS09STR 

BPS09PSU; 

DESIGN = WR 

WEIGHT WTC000; 

NEST BPS09STR 
BPS09PSU; 

Stata 

svyset BPS09PSU [pweight = 
WTA000], 
strata (BPS09STR) 
vce(LINEARIZED) 

svyset BPS09PSU [pweight = 
WTB000], 
strata (BPS09STR) 
vce(LINEARIZED) 

svyset BPS09PSU; [pweight 
= WTC000], 
strata (BPS09STR) 
vce(LINEARIZED) 

SAS survey analysis 
procedures 

VARMETHOD = JACKKNIFE 
WEIGHT WTA000; 

STRATA BPS09STR; 
CLUSTER BPS09PSU; 

VARMETHOD = JACKKNIFE 
WEIGHT WTB000; 

STRATA BPS09STR; 
CLUSTER BPS09PSU; 

VARMETHOD = JACKKNIFE 
WEIGHT WTC000; 

STRATA BPS09STR; 
CLUSTER BPS09PSU; 

IBM SPSS complex 
samples 

Method: WR 

Weight: WTA000 

Strata: BPS09STR 

PSU: BPS09PSU 

Method: WR 

Weight: WTB000 

Strata: BPS09STR 

PSU: BPS09PSU 

Method: WR 

Weight: WTC000 

Strata: BPS09STR 

PSU: BPS09PSU 

Taylor series variance estimation (without replacement) 

Variance estimation strata, FB09STR, FB09PSU, 

PSU, SSU, and count FB09SSU, and 

variables B9PSUCNT 

FB09STR, FB09PSU, 
FB09SSU, and 
B9PSUCNT 

FB09STR, FB09PSU, 
FB09SSU, and 
B9PSUCNT 


Software: statements, parameters, and keywords for Taylor series variance estimation (without replacement) 


SUDAAN 


Stata 


Bootstrap variance estimation 
Replicate weight variables 


DESIGN = WOR 

WEIGHT WTA000; 

NEST FB09STR FB09PSU 
FB09SSU; 

TOTCNT B9PSUCNT 
_minus1 zero_; 

svyset FB09PSU 
[pw=WTA000], 
strata(FB09STR) 
fpc(PSUCOUNT) || 
FB09SSU, 
vce(LINEARIZED) 


WTA001 -WTA200 


DESIGN = WOR 

WEIGHT WTB000; 

NEST FB09STR FB09PSU 
FB09SSU; 

TOTCNT B9PSUCNT 
_minus1 zero_; 

svyset FB09PSU 
[pw=WTB000], 
strata(FB09STR) 
fpc(PSUCOUNT) || 
FB09SSU, 
vce(LINEARIZED) 


WTB001 - WTB200 


DESIGN = WOR 

WEIGHT WTC000; 

NEST FB09STR FB09PSU 
FB09SSU; 

TOTCNT B9PSUCNT 
_minus1 zero_; 

svyset FB09PSU 
[pw=WTB000], 
strata(FB09STR) 
fpc(PSUCOUNT) || 
FB09SSU, 
vce(LINEARIZED) 


WTC001 -WTC200 


DESIGN = BRR 
WEIGHT WTC000; 
REPWGT WTC001 - 
WTC200; 


Software: statements, parameters, and keywords for BRR variance estimation 


SUDAAN 


DESIGN = BRR 
WEIGHT WTA000; 
REPWGT WTA001 
WTA200; 


DESIGN = BRR 
WEIGHT WTB000; 
REPWGT WTB001 - 
WTB200; 


See notes at end of table. 
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Table 44. Use of analysis weights, replicate weights, and variance estimation strata and primary 
sampling unit (PSU) variables available from BPS:04/09 in selected survey data 
analysis software: 2009 — Continued 


Survey data analysis software 

BPS: 04/09 study 
respondents; NPSAS:04 and 
BPS:04/09 study respondents 

Study respondents to all 
three of NPSAS:04, 

BPS:04/06, and BPS:04/09 

Students with at least one 
transcript at BPS:04/09 


svyset [pweight=WTA000], 

svyset [pweight=WTB000], 

svyset [pweight=WTC000], 

Stata 

brrweight(WTA001 - 

brrweight(WTB001 - 

brrweight(WTC001 - 


WTA200) vce(BRR) 

WTB200) vce(BRR) 

WTC200) vce(BRR) 


VARMETHOD = BRR 

VARMETHOD = BRR 

VARMETHOD = BRR 

SAS survey data 

WEIGHT WTA000; 

WEIGHT WTB000; 

WEIGHT WTC000; 

analysis procedures 

REPWEIGHTS WTA001 - 

REPWEIGHTS WTB001 - 

REPWEIGHTS WTC001 - 


WTA200; 

WTB200; 

WTC200; 


Method: BRR 

Method: BRR 

Method: BRR 

\/\/pc\/rir 

Full sample weight: WTA000 

Full sample weight: WTB000 

Full sample weight: WTC000 

v vco v a i 

Replicates: WTA001 - 

Replicates: WTB001 - 

Replicates: WTC001 - 


WTA200 

WTB200 

WTC200 


Note: The survey data analysis software specifications are given for the following versions of the software packages: SUDAAN 
10.0.1, Stata 12, SAS 9.2, IBM SPSS complex samples 20, and WesVar 4.3. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

6.2.2 Bootstrap Replicate Weights 

The variance estimation strategy that was chosen for BPS:04/ 09 is the same as that used for 
NPSAS:04 and BPS:04/ 06 and satisfies the following requirements: 

• recognition of variance reduction due to stratification at all stages of sampling; 

• recognition of the effects of unequal weighting; 

• recognition of possible increased variance due to sample clustering; 

• recognition of the effects of weight adjustments for nonresponse and for calibration of 
selected total estimates to known external totals; 

• satisfactory properties for estimating variances of nonlinear statistics and quantiles (such 
as the median) as well as for linear statistics; 

• ability to apply finite population corrections at the institution stage of sampling and 
reflect the reduction in variance due to the high sampling rates in some first-stage 
sampling strata; and 

• ability to test hypotheses about students based on normal distribution theory by 
ignoring the finite population corrections at the student level of sampling. 

Commonly applied bootstrap variance estimation techniques satisfy the first five 
requirements. To meet the last two requirements as well, a methodology developed by Kaufman 
(2004) was applied, allowing for finite population correction factors at two stages of sampling. The 
application of Kaufman’s method, which was used for NPSAS:04 and also BPS:04/ 06 and 
BPS:04/ 09, incorporated the finite population correction factor at the first stage only, where 
sampling fractions were generally high. At the second stage, where the sampling fractions were 
generally low, the finite population correction factor was set to 1.00. 

The Kaufman methodology was used to develop a vector of bootstrap sample weights that 
was added to the analysis file. These weights are zero for units not selected in a particular bootstrap 
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sample; weights for other units are inflated for the bootstrap subsampling. The initial analytic 
weights for the complete sample are also included for the purposes of computing the desired 
estimates. The vector of replicate weights allows for computing additional estimates for the sole 
purpose of estimating a variance. Assuming B sets of replicate weights, the variance of any estimate, 

0 , can be estimated by replicating the estimation procedure for each replicate and computing a 
simple variance of the replicate estimates, as follows: 

m-of 

var(6?) = — — , 

n 


where 0^ is the estimate based on the £th replicate weight and B is the total number of sets of 
replicate weights. Once the replicate weights are provided, this estimate can be produced by most 
survey software packages (e.g., SUDAAN [RTI International 2008] computes this estimate by 
invoicing the DESIGN = BRR option). 

The number of replicate weights was set at 200 for each of the NPSAS:04, BPS:04/ 06, and 
BPS:04/ 09 analysis weights, based on work that showed that this number of replicates has desirable 
properties for variance estimation in regression analyses. For the 200 replicate weights included on 
the weights file, both the nonresponse adjustment and the calibration process was repeated so that 
the variance of survey estimates would include the variability due to the weight adjustments. The 
analysis and replicate weights that are available on the weights file for BPS:04/ 09 are the following: 

Type of respondents Analysis weight Replicate weights 

Study respondents WTA000 WTA001-WTA200 

Panel respondents WTB000 WTB001-WTB200 

Cases with transcript data WTC000 WTC001-WTC200 

6.3 Overall Weighted and Unweighted Response Rates 

The overall BPS:04/ 09 response rate is an estimate of the proportion of the study 
population directly represented by the respondents. Because the BPS:04/ 09 study includes a 
subsample of NPSAS:04 nonrespondents, the overall BPS:04/09 response rate is the product of the 
NPSAS:04 institution-level response rate times the BPS:04/ 09 student-level study response rate. 
Therefore, the overall BPS:04/ 09 response rates can only be estimated directly for defined 
institutional characteristics. 

Table 45 gives the unweighted and weighted NPSAS:04 base-year institution and 
BPS:04/ 09 student response rate components by type of institution. The types of student 
respondents included in table 45 are the following: 

• BPS:04/ 09 study respondents; 

• BPS:04/ 09 interview respondents; 

• Panel respondents (i.e., study respondents to all three of NPSAS:04, BPS:04/06, and 
BPS:04/09); and 

• BPS:04/09 transcript respondents (i.e., cases with any transcript data). 
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Table 45. Unweighted and weighted NPSAS:04 institution response rates and BPS:04/09 student 
study, interview, panel, and transcript response rates, by type of institution: 2009 


Institution response 
rate 

Eligible 

sample 

size 

Respondents 

Response rate 

Overall response rate 

Type of institution (base year) 

Un- 
weighted Weighted 

Un- 
weighted Weighted 

Un- 

weighted 

Weighted 

BPS:04/09 study respondents 

Total 

83.5 

80.0 

18,540 

16,680 

90.0 

89.2 

75.1 

71.3 

Public 









Less-than-2-year 

76.6 

74.3 

540 

430 

79.3 

79.0 

60.7 

58.7 

2-year 

85.4 

77.6 

6,310 

5,570 

88.3 

87.7 

75.4 

68.0 

4-year non-doctorate-granting 

85.1 

70.3 

1,690 

1,590 

94.3 

93.4 

80.2 

65.7 

4-year doctorate-granting 

86.3 

87.1 

3,070 

2,990 

97.3 

96.7 

84.0 

84.3 

Private nonprofit 









Less-than-4-year 

89.0 

92.6 

530 

440 

83.4 

85.5 

74.3 

79.2 

4-year non-doctorate-granting 

81.9 

78.1 

2,280 

2,190 

96.1 

94.4 

78.7 

73.7 

4-year doctorate-granting 

77.7 

80.8 

1,520 

1,490 

98.0 

97.5 

76.2 

78.8 

Private for-profit 









Less-than-2-year 

84.0 

82.3 

1,450 

1,070 

74.2 

74.3 

62.3 

61.1 

2 years or more 

84.4 

88.2 

1,150 

900 

78.7 

77.9 

66.4 

68.7 

BPS:04/09 interview respondents 

Total 

83.5 

80.0 

18,540 

15,160 

81.8 

80.2 

68.3 

64.1 

Public 









Less-than-2-year 

76.6 

74.3 

540 

420 

77.0 

77.5 

59.0 

57.6 

2-year 

85.4 

77.6 

6,310 

5,010 

79.4 

78.1 

67.8 

60.6 

4-year non-doctorate-granting 

85.1 

70.3 

1,690 

1,430 

84.6 

82.6 

72.0 

58.1 

4-year doctorate-granting 

86.3 

87.1 

3,070 

2,720 

88.4 

87.2 

76.3 

76.0 

Private nonprofit 









Less-than-4-year 

89.0 

92.6 

530 

400 

75.7 

79.9 

67.4 

74.0 

4-year non-doctorate-granting 

81.9 

78.1 

2,280 

1,990 

87.2 

84.2 

71.4 

65.8 

4-year doctorate-granting 

77.7 

80.8 

1,520 

1,340 

88.4 

86.8 

68.7 

70.1 

Private for-profit 









Less-than-2-year 

84.0 

82.3 

1,450 

1,030 

70.8 

70.6 

59.5 

58.1 

2 years or more 

84.4 

88.2 

1,150 

830 

72.5 

72.2 

61.2 

63.6 

BPS:04/09 panel respondents 

Total 

83.5 

80.0 

18,540 

16,120 

87.0 

85.7 

72.6 

68.6 

Public 









Less-than-2-year 

76.6 

74.3 

540 

380 

70.6 

71.3 

54.0 

53.0 

2-year 

85.4 

77.6 

6,310 

5,360 

85.0 

83.6 

72.6 

64.9 

4-year non-doctorate-granting 

85.1 

70.3 

1,690 

1,560 

92.7 

91.7 

78.9 

64.5 

4-year doctorate-granting 

86.3 

87.1 

3,070 

2,950 

95.9 

95.0 

82.7 

82.7 

Private nonprofit 









Less-than-4-year 

89.0 

92.6 

530 

420 

78.7 

80.3 

70.1 

74.3 

4-year non-doctorate-granting 

81.9 

78.1 

2,280 

2,170 

95.0 

92.1 

77.8 

71.9 

4-year doctorate-granting 

77.7 

80.8 

1,520 

1,480 

97.0 

96.5 

75.4 

78.0 

Private for-profit 









Less-than-2-year 

84.0 

82.3 

1,450 

950 

65.8 

64.6 

55.3 

53.2 

2 years or more 

84.4 

88.2 

1,150 

860 

74.7 

74.4 

63.0 

65.6 

BPS:04/09 transcript respondents 

Total 

83.5 

80.0 

18,540 

16,960 

91.5 

91.4 

76.4 

73.1 

Public 









Less-than-2-year 

76.6 

74.3 

540 

400 

74.8 

74.7 

57.3 

55.5 

2-year 

85.4 

77.6 

6,310 

5,840 

92.6 

91.7 

79.0 

71.1 

4-year non-doctorate-granting 

85.1 

70.3 

1,690 

1,540 

91.0 

90.4 

77.5 

63.6 

4-year doctorate-granting 

86.3 

87.1 

3,070 

2,940 

95.7 

95.9 

82.6 

83.6 


See notes at end of table. 
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Table 45. Unweighted and weighted NPSAS:04 institution response rates and BPS:04/09 student 
study, interview, panel, and transcript response rates, by type of institution: 2009 — 
Continued 



Institution response 
rate 

Eligible 


Response rate 

Overall response rate 

Type of institution (base year) 

Un- 

weighted 

Weighted 

sample 

size 

Respondents 

Un- 

weighted 

Weighted 

Un- 

weighted 

Weighted 

Private nonprofit 









Less-than-4-year 

89.0 

92.6 

530 

480 

90.0 

89.0 

80.1 

82.4 

4-year non-doctorate-granting 

81.9 

78.1 

2,280 

2,210 

96.8 

96.9 

79.3 

75.7 

4-year doctorate-granting 

77.7 

80.8 

1,520 

1,440 

94.5 

95.1 

73.5 

76.9 

Private for-profit 









Less-than-2-year 

84.0 

82.3 

1,450 

1,080 

74.3 

74.0 

62.4 

60.9 

2 years or more 

84.4 

88.2 

1,150 

1,030 

90.0 

89.7 

75.9 

79.2 


NOTE: Detail may not sum to totals because of rounding. Base-year institution response rates were obtained from the 2003-04 
National Postsecondary Student Aid Study (NPSAS:04) Methodology Report (Cominole et al. 2006, table 8, p. 48). Overall 
response rates are the product of the NPSAS:04 and BPS:04/09 response rates. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


The institution-level response rates shown in table 45 are the percentage of institutions that 
provided sufficient data to select the NPSAS:04 student-level sample; these rates are presented and 
discussed in the NPSAS:04 Full-scale Methodology Report (Cominole et al. 2006, table 8, p. 48). 
Only the weighted response rates can be interpreted as estimates of the proportion of the 
BPS:04/ 09 population that is directly represented by the respondents. 

Table 45 shows that, across all types of institutions, 89 percent of the eligible BPS:04/ 09 
sample were study respondents. The rate varied from 74 percent to 97 percent, by type of 
institution. The overall weighted response rate, incorporating the NPSAS:04 base-year institution 
response rate, was 71 percent. The study analysis weight described in section 6.1 (WTA000) was 
developed to compensate for the potentially biasing effects of study nonresponse. 

This table also shows that approximately 80 percent of the eligible sample responded to the 
BPS:04/ 09 interview. All of the study respondents, including those who did not respond to the 
interview, have enrollment data from either the student interview or another source, such as their 
transcript data or the National Student Clearinghouse (NSC) StudentTracker. Enrollment data 
account for about half of the BPS:04/ 09 data, and imputation was used to compensate for 
nonresponse for the remaining items for those study respondents who did not provide interview 
data. A separate weight for analyzing interview respondents is not provided because imputation was 
used to reduce nonresponse bias due to interview nonresponse. The weighted response rate for 
those providing interviews was 80 percent and varied from 71 percent to 87 percent, by type of 
institution. These interview rates are provided for users of the data who may wish to analyze the 
cases without imputed data. A related bias analysis is provided in section 6.4.2. 

Overall, 86 percent of the sample were panel respondents. To be a panel respondent, the 
student needed to have sufficient data from either the interview or another source at NPSAS:04, 
have enrollment data from either the BPS:04/ 06 interview or another source, and have enrollment 
data from either the BPS:04/09 interviewer or another source. This rate varied, by type of 
institution, from 65 percent to 97 percent. The weight variable WTB000 was developed for 
analyzing the NPSAS:04-BPS:04/06-BPS:04/09 panel respondents. 

Table 45 also provides weighted response rates for the transcript data collection 
component. Overall, at least one transcript was collected from 91 percent of the eligible students. 
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This varied, by type of institution, from 74 percent to 97 percent. An analysis weight (the weight 
variable WTC000) was developed for analyzing students with any transcript data. 

Section 6.4.2 analyzes the potential bias due to unit nonresponse and the effect the weight 
adjustments had in reducing the bias. 

6.4 Accuracy of Estimates 

The accuracy of survey statistics is affected by both random and nonrandom errors. 
Random errors reduce the precision of survey estimates, while nonrandom errors result in bias (i.e., 
estimates that do not converge to the tme population parameter as the sample size increases 
without limit). 

The sources of error in a survey are often dichotomized as sampling and nonsampling 
errors. Sampling error refers to the error that occurs because the survey is based on a sample of 
population members rather than the entire population. All other types of errors are nonsampling 
errors , including survey nonresponse (because of inability to contact sampling members, their refusal 
to participate in the study, etc.) and measurement errors , such as the errors that occur because the 
intent of survey questions was not clear to the respondent, because the respondent had insufficient 
knowledge to answer correctly, or because the data were not captured correctly (e.g., because of 
recording, editing, or data entry errors). 

The sampling errors are primarily random errors for well-designed surveys such as 
NPSAS:04, BPS:04/06, and BPS:04/09. However, nonrandom errors can occur if the sampling 
frame does not provide complete coverage of the target population. The BPS:04/09 survey 
instrument and data collection procedures were subjected to thorough development and testing to 
minimize nonsampling errors because these errors are difficult to quantify and are likely to be 
nonrandom errors. 

In this section, sampling errors and design effects for some BPS:04/ 09 estimates are 
presented for a variety of domains; these sampling errors and design effects are computed using 
each of the analysis weights that was constmcted for analyzing the BPS:04/ 09 student and 
transcript data. 

Next, the results of analyses comparing BPS:04/09 nonrespondents and respondents using 
characteristics known for both nonrespondents and respondents are presented. An analysis of 
nonresponse bias is presented at both the student level and the item level. 

6.4.1 Measures of Precision: Standard Errors and Design Effects 

The survey design effect for a statistic is defined as the ratio of the design-based variance 
estimate divided by the variance estimate that would have been obtained from a simple random 
sample of the same size. The design effect is often used to measure the effects that sample design 
features have on the precision of survey estimates. For example, stratification tends to decrease the 
variance, but multistage sampling and unequal sampling rates usually increase the variance. Weight 
adjustments for nonresponse (performed to reduce nonresponse bias) and calibration often 
increase the variance because they can increase the weight variation. Because of these factors, 
estimates from most complex multistage sampling designs such as BPS:04/09 have design effects 
greater than 1.0. That is, the design-based variance is larger than the simple random sample 
variance. 
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Specifically, the survey design effect for a given estimate, 0 , is defined as 

design 

Var„(0) ' 

The square root of the design effect can also be expressed as the ratio of the standard 
errors, or 

De/m) = SEdesisn ^ . 

SKA 9 ) 

In appendix L, design effect estimates are presented for important survey domains to 
summarize the effects of stratification, multistage sampling, unequal probabilities of selection, and 
the weight adjustments. These design effects were estimated using SUDAAN and the bootstrap 
variance estimation procedure described in section 6.2.2. If an analysis of BPS:04/ 09 data must be 
performed without using one of the software packages for analysis of complex survey data, the 
design effect tables in appendix L can be used to make approximate adjustments to the standard 
errors of survey statistics computed using the standard software packages that assume simple 
random sampling designs. 

Large design effects imply large standard errors and relatively poor precision. Small design 
effects imply small standard errors and good precision. In general terms, a design effect under 2.0 is 
low, 2.0 to 3.0 is moderate, and above 3.0 is high. Moderate and high design effects often occur in 
complex surveys such as BPS:04/09, and the design effects in appendix L are consistent with those 
in past BPS studies. Unequal weighting causes large design effects and is often as a result of 
nonresponse and poststratification adjustments. However, in BPS:04/09 (as in BPS:04/06 and 
NPSAS:04), the unequal weighting is also due to the sample design, different sampling rates 
between institution strata, different sampling rates between student strata, and subsampling of the 
nonrespondents that were included in BPS:04/ 06. 

6.4.2 Measure of Bias 

The bias in an estimated mean based on respondents, y R , is the difference between this 

mean and the target parameter, n, that is, the mean that would be estimated if a complete census of 
the target population was conducted and everyone responded. This bias can be expressed as 
follows, where E\y R ) is the expected value of the mean based on respondents over repeated 
samples: 

B{y R )= E{y R )~ K . 

The estimated mean based on nonrespondents, y NR , can be computed if data for the 

particular variable are available for most of the nonrespondents. The tme target parameter, n, can 
be estimated for these variables as follows: 

^ = (i -n)y R + ny m > 

where T) is the weighted unit (or item) nonresponse rate. For the variables that are from the frame, 
rather than from the sample, n can be estimated without sampling error. The bias can then be 
estimated as follows: 


De/m 


( 0 ) 
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B(y R ) = y R -n 

or, equivalently, 

B{y R ) = n{y R -y NR )- 

This formula shows that the estimate of the nonresponse bias is the difference between the 
mean for respondents and nonrespondents multiplied by the weighted nonresponse rate. 

Nonresponse bias analysis was conducted when the response rate at any level (institutions, 
students, items) was below 85 percent. 22 Institution nonresponse bias was performed as a part of 
NPSAS:04 and is described in the NPSAS:04 Full-scale Methodology Report (Cominole et al. 

2006). A student nonresponse bias analysis and an item nonresponse bias analysis were also 
performed for BPS:04/06. The remainder of this section summarizes the unit and item 
nonresponse analyses that were conducted for BPS:04/ 09. 

Unit nonresponse bias analysis and relative bias. Unit nonresponse bias analyses were 
conducted for the following sets of respondents: 

• BPS:04/09 study respondents versus the full set of cases eligible for BPS:04/09 (study 
respondents and study nonrespondents), before and after the weight adjustment that 
resulted in the BPS:04/09 study weight (WTA000); 

• BPS:04/ 09 interview respondents versus the full set of cases eligible for BPS:04/ 09 
(interview respondents and interview nonrespondents), using both the BPS:04/ 06 
weight and the BPS:04/09 study analysis weight (WTA000); 

• panel respondents (i.e., cases who were study respondents to all three of NPSAS:04, 
BPS:04/ 06, and BPS:04/ 09) versus the full set of cases eligible for BPS:04/ 09 (panel 
respondents and cases included in the sample who were not study respondents to all 
three waves), using the BPS:04/06 weight and then again using the panel weight 
WTB000; and 

• transcript respondents versus the full set of cases eligible for BPS:04/ 09 (transcript 
respondents and transcript nonrespondents), before and after the transcript weight 
adjustment that resulted in the BPS:04/ 09 student transcript weight WTC000. 

Tables in appendix M give the estimates for respondents before the weight adjustment, 
nonrespondents, and the full sample. Estimates are also given for the respondents after weight 
adjustment. The nonresponse bias was estimated for variables obtained from the sampling frame 
and from the NPSAS:04 data collection that are known for both respondents and nonrespondents. 
In all of the tables, the bias was estimated as follows. First, the percentage distribution was obtained 
for the respondents using the weight. 23 Next, the percentage distribution was obtained for the 
overall sample using the BPS:04/ 06 analysis weight. Then, the bias was estimated as the difference 
in the percentages. Statistical tests of the bias were also computed using bootstrap estimates of the 
standard errors, and the tables in appendix M indicate when the bias is statistically different from 
zero. 


22 See NCES Statistical Standards (U.S. Department of Education, Institute of Education Sciences, National Center for Education 
Statistics 2003) for a discussion of nonresponse bias analysis. 

23 For categorical variables, the y bars in the equations are the weighted percentages in each category of the variables (for example, 
for the gender variable, the y bars are the percentage that are male, or the percentage that are female). 
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It is also informative to compare the distributions of the respondents and nonrespondents, 
and the tables in appendix M include columns that give the weighted distributions of respondents 
and nonrespondents. From the above formulas, the bias prior to the weight adjustment can also be 
obtained as the nonresponse rate multiplied by the difference between respondents and 
nonrespondents. When the bias before the weight adjustment is statistically significant, the 
differences between the respondent and nonrespondent distributions are almost always statistically 
significant. Similarly, when the differences between the respondent and nonrespondent 
distributions are statistically significant, the bias is also statistically significant. When one is 
statistically significant but not the other, the p values are very close to .05. The p values are not 
identical because of the sampling error associated with the nonresponse rate. For this reason, 
separate columns that give statistical tests are not provided in appendix M for comparing the 
respondent and nonrespondent percentages. 

The variables (and number of categories of each variable) that were used in the analyses for 
all sample members are the following: 

• type of institution in the base year (9 categories); 

• region (9 categories); 

• Central Processing System (CPS) match at the base year (yes/ no) (2 categories); 

• applied for federal aid (yes /no) (2 categories), 

• Pell Grant recipient (yes/no) (2 categories); 

• total Pell Grant amount received (4 categories); 

• Stafford Loan recipient (yes/no) (2 categories); 

• total Stafford Loan amount received (4 categories); 

• base-year institution undergraduate enrollment (5 categories) 

• age at base year (5 categories); 

• high school graduation year (4 categories); 

• dependency status at base year (2 categories); 

• income level at base year (15 categories); 

• race/ethnicity (5 categories); 

• gender (2 categories); 

• marital status at base year (3 categories); and 

• citizenship status at base year (3 categories). 

The nonresponse bias was estimated for the above variables and tested to determine if the 
bias was significant at the 5 percent level. The tests are reported to be statistically significant if the 
p value is less than .05. Results are given in appendix M for all institutions combined and by type of 
institution. 
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Table 46 summarizes the results of the bias analysis for study respondents. This analysis 
estimated the bias prior to the weight adjustment using the BPS:04/ 06 weight or, equivalently, 
compared the BPS:04/09 study respondents and study nonrespondents. Tables in appendix M give 
the estimated bias before and after the weight adjustment that resulted in the study analysis weight 
WTA000. Appendix M also contains tables for all students and for each of the nine institution 
types. 

Table 46. Summary of student study nonresponse bias before and after weight adjustments, by 
type of institution: 2009 


Significant variable 

Nonresponse bias statistics categories 


Type of institution 

Mean 

Median 

Minimum 

Maximum 

Percent 

Number 

Total 







Before weight adjustments 

0.62 

0.43 

0.00 

2.85 

67.9 

53 

After weight adjustments 

0.05 

0.02 

0.00 

0.32 

1.3 

1 

Public less-than-2-year 







Before weight adjustments 

0.82 

0.40 

0.02 

4.14 

11.8 

8 

After weight adjustments 

0.95 

0.47 

0.01 

4.61 

13.2 

9 

Public 2-year 







Before weight adjustments 

0.30 

0.17 

0.00 

1.46 

24.6 

17 

After weight adjustments 

0.14 

0.11 

0.00 

0.62 

2.9 

2 

Public 4-year non-doctorate-granting 







Before weight adjustments 

0.40 

0.26 

0.01 

1.87 

14.7 

10 

After weight adjustments 

0.23 

0.22 

0.00 

0.71 

0.0 

0 

Public 4-year doctorate-granting 







Before weight adjustments 

0.24 

0.20 

0.00 

1.01 

35.8 

24 

After weight adjustments 

0.13 

0.09 

0.00 

0.45 

0.0 

0 

Private nonprofit less-than-4-year 







Before weight adjustments 

0.57 

0.49 

0.00 

2.12 

0.0 

0 

After weight adjustments 

0.64 

0.38 

0.00 

2.32 

0.0 

0 

Private nonprofit 4-year non-doctorate- 







granting 

Before weight adjustments 

0.49 

0.32 

0.00 

1.90 

30.4 

21 

After weight adjustments 

0.24 

0.23 

0.01 

0.81 

0.0 

0 

Private nonprofit 4-year doctorate-granting 







Before weight adjustments 

0.17 

0.09 

0.00 

0.74 

13.0 

9 

After weight adjustments 

0.14 

0.11 

0.00 

0.72 

0.0 

0 

Private for-profit less-than-2-year 







Before weight adjustments 

0.77 

0.63 

0.01 

3.68 

23.2 

16 

After weight adjustments 

0.40 

0.28 

0.01 

1.61 

5.8 

4 

Private for-profit 2 years or more 







Before weight adjustments 

0.72 

0.45 

0.01 

2.99 

2.9 

2 

After weight adjustments 

0.75 

0.48 

0.01 

3.12 

2.9 

2 


NOTE: The statistics “before weight adjustments” were based on using the BPS:04/06 analysis weight. The statistics “after weight 
adjustments” were based on using the BPS:04/09 study weight WTA000. The percents are based on the total number of variable 
categories that have nonzero respondents and nonrespondents (usually 78 for the total and 69 by type of institution). SOURCE: 
U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students Longitudinal 
Study (BPS:04/09). 
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The NCES Statistical Standards (U.S. Department of Education [ED], Institute of Education 
Sciences [TES], National Center for Education Statistics [NCES] 2003) requires a bias analysis for 
any stage of a sample with a response rate less than 85 percent. From table 45, the weighted 
BPS:04/ 09 study response rate was less than 85 percent for students whose base-year institution 
was public less-than-2-year, private for-profit less-than-2-year, and private for-profit 2 years or 
more. Table 46 shows that the bias was generally reduced across all of the types of institutions. In 
particular, for students in the private for-profit less-than-2-year institutions, 23 percent of the 
variable categories had statistically significant bias before the weight adjustment, and 6 percent had 
statistically significant bias after the weight adjustment. For students in the private for-profit 2 years 
or more institutions, the percent of the variable categories with statistically significant bias was the 
same before and after weight adjustment (3 percent). One exception is for students in the public 
less-than-2-year institutions, where the percentage of variable categories with statistically significant 
bias increased from 12 percent to 13 percent using the adjusted weight. This type of institution has 
a relatively small sample size, however, and the actual bias in the variables remained about the 
same. 

An analysis of bias was also conducted for the interview respondents. As noted earlier, 
some cases classified as study respondents have enrollment data from sources other than the 
BPS:04/ 09 interview, but do not have BPS:04/ 09 interview data. The BPS:04/ 09 data file contains 
imputed variables for the interview data and some derived variables for those study respondents 
who were interview respondents. Stochastic imputation was used to reduce the bias due to 
interview nonresponse. A separate analysis weight was not constmcted that adjusted solely for the 
BPS:04/ 09 interview nonresponse. However, for users who might consider analyzing only the 
nonimputed data using the BPS:04/ 09 study weight, an analysis of the potential bias was 
conducted. 

Tables in appendix M compare BPS:04/09 interview respondents and nonrespondents, 
using the BPS:04/ 09 study weight. From table 45, the student interview response rates are less than 
85 percent for the full sample, and for students in all types of institutions except the public 4-year 
doctorate-granting and the private nonprofit 4-year doctorate-granting. Table 47 summarizes the 
bias among interview respondents using the BPS:04/ 06 weight and also using the BPS:04/ 09 study 
weight. The percentage of variable categories with statistically significant bias is usually smaller 
when the BPS:04/ 09 student study weight is used for analysis compared to when the BPS:04/ 06 
analysis weight is used. Exceptions are the public less-than-2-year institutions, the public 2-year 
institutions, and the private for-profit 2 years or more institutions. For these types of institutions, 
the actual bias is fairly similar using the two weights. This analysis indicates that the preferred 
weight for the data users who which to analyze the interview respondents with nonimputed data is 
the BPS:04/ 09 study weight. However, users of the data should be aware that the adjustments 
applied to develop the study weight do not adjust for all of the interview nonresponse and some 
biases may still exist in the estimates. The actual estimated biases are all less than 5 percent for the 
variables considered in this analysis. However, across all types of institutions, almost one-third of 
the variable categories have statistically significant bias when the study weight is used for analyzing 
the interview respondents. 
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Table 47. Summary of student nonresponse bias for interview respondents when analyzed using 
the study weight, by type of institution: 2009 


Significant variable 

Nonresponse bias statistics categories 


Type of institution 

Mean 

Median 

Minimum 

Maximum 

Percent 

Number 

Total 

Using BPS:04/06 weight 

0.58 

0.40 

0.00 

2.31 

57.7 

45 

Using BPS:04/09 study weight 

0.25 

0.18 

0.00 

0.88 

32.1 

25 

Public less-than-2-year 

Using BPS:04/06 weight 

0.91 

0.44 

0.03 

4.24 

8.8 

6 

Using BPS:04/09 study weight 

1.03 

0.57 

0.00 

4.60 

19.1 

13 

Public 2-year 

Using BPS:04/06 weight 

0.32 

0.19 

0.00 

1.62 

13.0 

9 

Using BPS:04/09 study weight 

0.34 

0.24 

0.00 

1.29 

17.4 

12 

Public 4-year non-doctorate-granting 

Using BPS:04/06 weight 

0.59 

0.38 

0.01 

2.03 

16.2 

11 

Using BPS:04/09 study weight 

0.38 

0.30 

0.00 

1.22 

4.4 

3 

Public 4-year doctorate-granting 

Using BPS:04/06 weight 

0.42 

0.31 

0.01 

1.40 

26.9 

18 

Using BPS:04/09 study weight 

0.30 

0.24 

0.01 

0.93 

6.0 

4 

Private nonprofit less-than-4-year 

Using BPS:04/06 weight 

0.72 

0.63 

0.07 

2.65 

0.0 

0 

Using BPS:04/09 study weight 

0.88 

0.47 

0.01 

3.93 

0.0 

0 

Private nonprofit 4-year non-doctorate- 
granting 

Using BPS:04/06 weight 

0.83 

0.56 

0.01 

3.17 

37.7 

26 

Using BPS:04/09 study weight 

0.48 

0.34 

0.02 

2.15 

8.7 

6 

Private nonprofit 4-year doctorate-granting 

Using BPS:04/06 weight 

0.41 

0.19 

0.00 

2.37 

11.6 

8 

Using BPS:04/09 study weight 

0.38 

0.26 

0.00 

1.71 

1.4 

1 

Private for-profit less-than-2-year 

Using BPS:04/06 weight 

0.69 

0.54 

0.01 

2.93 

21.7 

15 

Using BPS:04/09 study weight 

0.52 

0.35 

0.01 

1.94 

5.8 

4 

Private for-profit 2 years or more 

Using BPS:04/06 weight 

0.57 

0.37 

0.01 

3.00 

0.0 

0 

Using BPS:04/09 study weight 

0.77 

0.50 

0.01 

3.29 

2.9 

2 


The percents are based on the total number of variable categories that have nonzero respondents and nonrespondents (usually 78 
for the total and 69 by type of institution). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

An analysis was also conducted to compare the panel respondents and panel 
nonrespondents and to see if significant bias remains after the weight adjustment that resulted in 
the longitudinal or panel weight WTB000. From table 45, students in the following types of 
institutions had panel response rates less than 85 percent: public less-than-2-year, public 2-year, 
private nonprofit less-than-4-year, private for-profit less-than-2-year, and private for-profit 2 years 
or more. Tables in appendix M give detailed results of the analysis for each type of institution. The 
summary in table 48 shows that, prior to weight adjustment, nonresponse bias was significant for 
72 percent of the variable categories and, after weight adjustment, was significant for 5 percent of 
the variable categories. The weight adjustments resulted in a reduction of bias across all institution 
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types, except for students in private for-profit 2-year or more institutions where 3 percent of the 
variable categories had statistically significant bias before and after the weight adjustment and in 
private nonprofit less-than-4-year institutions where none of the variable categories had statistically 
significant bias before and after weight adjustment. 

Table 48. Summary of student NPSAS:04-BPS:04/06-BPS:04/09 nonresponse bias, by type of 
institution: 2009 


Significant variable 

Nonresponse bias statistics categories 


Type of institution 

Mean 

Median 

Minimum 

Maximum 

Percent 

Number 

Total 

Before weight adjustments 

0.78 

0.55 

0.02 

3.92 

71.8 

56 

After weight adjustments 

0.07 

0.02 

0.00 

0.51 

5.1 

4 

Public less-than-2-year 

Before weight adjustments 

0.95 

0.64 

0.04 

4.25 

7.4 

5 

After weight adjustments 

0.94 

0.61 

0.02 

4.25 

5.9 

4 

Public 2-year 

Before weight adjustments 

0.45 

0.27 

0.00 

2.28 

34.8 

24 

After weight adjustments 

0.21 

0.17 

0.00 

0.92 

4.3 

3 

Public 4-year non-doctorate-granting 

Before weight adjustments 

0.49 

0.36 

0.01 

2.36 

19.1 

13 

After weight adjustments 

0.35 

0.26 

0.00 

1.34 

5.9 

4 

Public 4-year doctorate-granting 

Before weight adjustments 

0.32 

0.21 

0.00 

1.53 

35.8 

24 

After weight adjustments 

0.20 

0.15 

0.00 

0.88 

6.0 

4 

Private nonprofit less-than-4-year 

Before weight adjustments 

0.78 

0.64 

0.01 

2.56 

0.0 

0 

After weight adjustments 

0.79 

0.51 

0.02 

3.64 

0.0 

0 

Private nonprofit 4-year non-doctorate- 
granting 

Before weight adjustments 

0.80 

0.63 

0.00 

3.55 

46.4 

32 

After weight adjustments 

0.39 

0.25 

0.01 

1.43 

5.8 

4 

Private nonprofit 4-year doctorate-granting 

Before weight adjustments 

0.26 

0.10 

0.00 

1.06 

11.6 

8 

After weight adjustments 

0.27 

0.19 

0.01 

0.98 

0.0 

0 

Private for-profit less-than-2-year 

Before weight adjustments 

0.72 

0.51 

0.00 

3.87 

13.0 

9 

After weight adjustments 

0.79 

0.52 

0.01 

3.29 

11.6 

8 

Private for-profit 2 years or more 

Before weight adjustments 

0.73 

0.47 

0.02 

3.08 

2.9 

2 

After weight adjustments 

0.92 

0.56 

0.01 

3.57 

2.9 

2 


NOTE: The statistics “before weight adjustments” were based on using the BPS:04/06 analysis weight. The statistics “after weight 
adjustments” were based on using the BPS:04/09 panel weight WTB000. The percents are based on the total number of variable 
categories that have nonzero respondents and nonrespondents (usually 78 for the total and 69 by type of institution). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 

From table 45, the percentage of BPS:04/09 students with transcript data was high overall 
(91 percent) and varied from 74 percent to 97 percent, by type of institution. Transcript data were 
available for fewer than 85 percent of students whose NPSAS institutions were public less-than-2- 
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year and private for-profit less-than-2-year. Tables in appendix M give the results of the analysis 
that compares transcript respondents and nonrespondents. Prior to the adjustment for 
nonresponse, across all institutions, 41 percent of the variable categories had statistically significant 
bias prior to the weight adjustment for nonresponse; the percentage was reduced to 4 percent after 
the adjustment (table 49). 

Table 49. Summary of student transcript nonresponse bias, by type of institution: 2009 


Significant variable 

Nonresponse bias statistics categories 


Type of institution 

Mean 

Median 

Minimum 

Maximum 

Percent 

Number 

Total 

Before weight adjustments 

0.37 

0.24 

0.00 

1.42 

41.0 

32 

After weight adjustments 

0.05 

0.02 

0.00 

0.42 

3.8 

3 

Public less-than-2-year 

Before weight adjustments 

1.40 

0.92 

0.05 

4.77 

0.0 

0 

After weight adjustments 

1.81 

1.20 

0.00 

6.31 

1.5 

1 

Public 2-year 

Before weight adjustments 

0.31 

0.12 

0.00 

1.23 

11.6 

8 

After weight adjustments 

0.20 

0.10 

0.00 

0.88 

8.7 

6 

Public 4-year non-doctorate-granting 

Before weight adjustments 

0.39 

0.23 

0.01 

1.52 

10.3 

7 

After weight adjustments 

0.41 

0.28 

0.00 

2.18 

1.5 

1 

Public 4-year doctorate-granting 

Before weight adjustments 

0.16 

0.07 

0.00 

1.04 

23.9 

16 

After weight adjustments 

0.13 

0.07 

0.00 

0.58 

3.0 

2 

Private nonprofit less-than-4-year 

Before weight adjustments 

1.16 

0.81 

0.01 

3.46 

1.4 

1 

After weight adjustments 

0.75 

0.60 

0.01 

4.14 

0.0 

0 

Private nonprofit 4-year non-doctorate- 
granting 

Before weight adjustments 

0.25 

0.16 

0.00 

1.49 

2.9 

2 

After weight adjustments 

0.30 

0.19 

0.00 

1.68 

0.0 

0 

Private nonprofit 4-year doctorate- 
granting 

Before weight adjustments 

0.24 

0.14 

0.00 

1.18 

21.7 

15 

After weight adjustments 

0.16 

0.09 

0.00 

1.40 

1.4 

1 

Private for-profit less-than-2-year 

Before weight adjustments 

0.68 

0.33 

0.00 

6.53 

18.8 

13 

After weight adjustments 

0.74 

0.48 

0.01 

3.75 

18.8 

13 

Private for-profit 2-year-or-more 

Before weight adjustments 

0.46 

0.31 

0.00 

2.89 

2.9 

2 

After weight adjustments 

0.48 

0.34 

0.00 

2.49 

1.4 

1 


NOTE: the statistics "before weight adjustments" were based on using the BPS:04/06 analysis weight. The statistics "after weight 
adjustments" were based on using the weight for analyzing the BPS:04/09 students with a transcript, WTC000. The percents are 
based on the total number of variable categories that have nonzero respondents and nonrespondents (usually 78 for the total and 
69 by type of institution). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004/09 Beginning Postsecondary Students 
Longitudinal Study (BPS:04/09). 


Item nonresponse bias analysis. When item response rates were less than 85 percent, the 
NCES Statistical Standards required that a nonresponse bias analysis be conducted. This analysis 
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was conducted on the data items collected in the BPS:04/ 09 interview based on study respondents 
and for variables derived from student transcript data collection. As shown in the equation below, 
item response rates (RR1) are calculated as the ratio of the number of respondents for whom an in- 
scope response was obtained (P for item x) to the number of respondents who are asked to answer 
that item. The number asked to answer an item is the number of unit-level respondents (I) minus 
the number of respondents with a valid skip for item x (IV). When an abbreviated questionnaire is 
used to convert refusals, the eliminated questions are treated as item nonresponse (ED, IES, NCES 
2003): 

RRI X =I X *(/ -V x ). 

Item response rates were computed using nonimputed data. Valid skips were later logically 
imputed to the follow-up items after the gate question was imputed (but these imputed skips count 
as missing for computing the response rate). Table J-l in appendix J lists the items from the 
BPS:04/ 09 interview along with the number of cases who were eligible to answer each item, and 
the weighted item response rates and nonresponse rates. The BPS:04/ 09 study weight (WTA000) 
was used to calculate the response rates. The nonresponse rate was also the same as the percentage 
of cases for which the item was imputed. The denominator of the item response rate included cases 
who were study respondents but who did not respond to the BPS:04/ 09 interview. As mentioned 
earlier, cases who did not respond to a gate item were treated as missing for the items within the 
gate. All but five of the 168 items listed in table J-l had item response rate less than 85 percent. 
These five items with response rate greater than 85 percent were GRENR09 (Current attendance in 
graduate school), SATMAJ09 (Satisfaction with choice of major), SEROLE09 (Primary role while 
working), SPSBOR09 (Spouse’s total student loan amount), and VOTEVE09 (Ever voted in a U.S. 
election). 

Table J-4 lists the derived variables from the transcript data along with the number of 
eligible cases and the weighted item response rates and nonresponse rates. The BPS:04/ 09 student 
transcript analysis weight (WTC000) was used to calculate the response rates. Of the 362 variables, 
all but 26 had a response rate greater than 85 percent. 

A nonresponse bias analysis was conducted for items with a weighted response rate less 
than 85 percent for all BPS:04/ 09 study respondents, and for derived transcript variables with 
weighted response rates less than 85 percent. The possibility of estimating the degree of bias 
depends on having some variables that reflect key characteristics of respondents and for which 
there is little or no missing data. The variables that were used (from the bulleted list above) are 
known for all BPS:04/ 09 study respondents. These variables are important to the study and are 
related to many of the items being analyzed for low item response rates. For the items with a 
weighted response rate less than 85 percent, the nonresponse bias prior to imputation was 
estimated for each of these characteristics that are known for respondents. 

Table M-41 in appendix M illustrates the estimated bias (prior to item imputation) for one 
item (GPA09 - Estimate of GPA) for BPS:04/ 09 study respondents. Similar computations were 
performed and tabulations were produced for each of the items. Table M-42 summarizes the results 
of the item nonresponse bias analysis for each of the items from the student interview, and gives 
the mean and median bias, the mean and median relative bias, and the percentage of the variables 
categories with statistically significant bias. Across the items, the percentage of variables with 
statistically significant bias ranged from 4 percent to 70 percent. Table M-43 gives the same analysis 
for the derived transcript items that have a weighted item response rate less than 85 percent. 
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Item imputation was used to fill in missing data for BPS:04/ 09 interview respondents and 
nonrespondents, as described in chapter 5. Item imputation was expected to reduce the bias due to 
item nonresponse, and was used instead of a separate weight adjustment for nonresponse for each 
item. All of the questionnaire items that are listed in table J-l were imputed using the imputation 
process described in chapter 5. 

A by-product of imputation was the reduction or elimination of item-level nonresponse 
bias. While item-level bias before imputation was measurable, after imputation it was not. As a 
result, how well an imputation procedure worked in reducing bias could not be directly evaluated. 
Instead, the before- and after-imputation item estimates were compared to determine whether the 
imputation significantly changed the biased estimates, thus suggesting a reduction in bias. Weighted 
estimates were computed using the nonimputed data (including only those cases who responded to 
the item) and also using the imputed data (including cases who responded to the item and also 
cases with imputed data for the item). Table J-2 gives the means before and after imputation for the 
continuous variables, and table J-3 gives the distributions before and after imputation for the 
categorical variables. These tables also give the difference between the pre-imputation and post- 
imputation estimates. The difference between the pre-and post-imputation estimates was 
statistically significant for 17 percent of the variables and variable categories. This suggests that 
imputation was only slightly successful in reducing the bias due to item nonresponse. 

Imputation was not performed for the items obtained from student transcript data. A 
weight, adjusted for students without any transcript data, was computed. Most of the variables that 
were derived from the transcript data have high item response rates (table J-4). 
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